2025-10-16 Board of Education's Audit Committee

001Okay. Um, hello everyone. It's 5:00 PM, actually a little after. So, I would like to call the audit committee meeting to order. I would will like to welcome each of you to the first audit committee meeting of the 2526 school year. We'll have a full agenda today. So, I would like to get us started with introductions. And I will start with myself. My name is Patty Sullivan and I am the PPS board member representing zone 3 and I'm chair of this audit committee. Oh, Andrea left just for a minute. Well, she'll be back by the time we get around. We'll start with you, Janice. >> Okay. My name is Janice Hansen and I'm the district's internal auditor. >> Hi, Michelle Morrison, chief financial officer. >> Hello, Caroline Zavikovski. I use she her pronouns and I'm

002a community member on the audit committee and I also have an auditing background as I work in the Moma County auditor's office. Okay, let's start with people who are on the screen there. Stephanie, you want to start or Stephanie, you want to introduce yourself? She's having trouble with the stuff. Okay, I'm going to go to Christie. >> Hi there. Can you hear me? >> Yes. >> Great. Um, I'm crossing the Broadway Bridge, so I'm very close after hitting very bad traffic trying to get over the hill from southwest Portland. Um, so I'll be there soon. I'm um the school board member from zone one. Really excited to be on the committee. grateful for the community membership as well and look forward to getting to know all of you once I get there in a few

003minutes. >> Okay, great. Okay. Um, Virginia, are you >> live? >> I'm alive. Hi, I'm not on the audit committee, but uh my name is Virginia. I'm the PPS board of director uh directors member for zone 5, and I saw that you have a packed agenda and um thought I would listen in. Looks like there's a lot going on, so I'm excited. Thanks for organizing this, Patty. >> Great. Welcome, >> Director Sullivan. >> Okay. And one more person here. >> Hi, I'm Jennifer Samuels. She her member of the audit committee. I'm a community member. Okay. Thank you. Welcome everybody. And um Andrea Shears, our senior legal counsel, had to zip out for a minute and she will be back, but I think we can start without her. Okay. Um, we do have a long agenda

004today and a lot of it will be me talking. So, I hope you don't get too tired of my voice. So, the next item on the agenda is the audit committee membership. The audit committee is unique from our other board committees in that we have two community members right there um who serve as exopicio members of this committee. I want to take a couple of minutes to draw attention to the value of having community members serving on the audit committee. The Council of Great City Schools recently issued a white paper on internal auditing that includes a section about the roles, responsibilities, and membership of audit committees. The white paper is posted for you in board books. To help facilitate this conversation, Janice printed a copy of the white paper for us and added a flag

005on the page that refers to community members. The second paragraph on page five. And if you flag that find it encourages everyone um states and I'm going to read it. Actually, I was going to have a board member read it, but they're not here. I'm the only one, so I'm going to read it. And the second paragraph refers to community rem um membership. To effectively oversee the internal audit function, the council recommends that a school district's audit committee be comprised of individuals independent of the school district and collectively have audit auditing, finance, information technology, risk management, and governance experience. An audit committee of experienced professionals with no authority over the school district's operations or decision-making processes can shield the internal audit staff from actual or perceived pressure that would compromise their objectivity and independence. So

006we really welcome you. Even the great city schools recognize how important you are. Um, the audit committee also welcomes up to two students to serve as exeicio members. Per board policy, students may be appointed to the audit committee by the student representative to the board and or at the discretion of the chair of the audit committee. I have reached out to our student representative Ian RTO and he will be letting students know about this opportunity. Even though we not ha we do not have any students for our first meeting, we will welcome students at any time during the year. The next item on the agenda is the status of the implementation of the contracts audit recommendations. One of the primary roles of the audit committee is to provide oversight and support for the office of

007internal performance audit. One of the ways the audit committee supports the work is to follow up on the status of the implementation of recommendations included in their audit reports. The PPS audit committee and internal performance audit charter states management will be responsible for reporting quarterly updates to the audit committee until recommendations are implemented. I have been included on some emailed communications with Dr. Adams. So, I'm aware that he and his team are working on preparing a report for the board that will include the review and evaluation of the goals identified in the contracts for the 2425 school year and identify the goals that were identified for the 2526 school year contracts. My understanding is that the information included in the report may be the evidence that the internal audit team is looking for to demonstrate

008the implementation of the contracts audit of the contracts audit recommendations number three and four. Dr. Adams did let me know that the report he is working on would not be ready to present to the audit committee today, but we did invite him to just give a kind of a general overview. However, he's not here. So, but he is planning to come to the October 28th board meeting um and uh we'll be able to answer our questions there. And also um if we need more information from Dr. Adam, we will invite him back to another um audit committee meeting. So the next item on the agenda is to discuss the status of the annual comprehensive financial report referred to as the ACFR audit and the audit of compliance with federal grant requirements referred to as the

009single audit. When I was at the PC last week, I bumped into Dan, the partner from TKW that is responsible for our audit. So, I happen to know that they are already on site working on the audit. I was planning just to mention that and talk about scheduling for the presentation of the results. However, I see that our CFO, wonderful Michelle Morrison, is to my right and she has provided us with a memo. So, Chief Morrison, would you like to go over the memo with us? >> Yes, thank you, Director Sullivan. So, we do have an issue with our uh single audit because each year the federal office of management and budget, OM releases an updated audit uh compliance guidelines known as the compliance supplement. And it was usually is released in May and it

010has not come out yet. And that are directions to our auditors on how to go about their testing of our federal programs. And so they are completing the work as if there hasn't been a lot of change. So they're going through their standard operating procedures as they would for our single audit um and drafting a report based on that. But it is may create some challenges for us if the um OM guidance comes out with new information because they'll have to come back and do some testing and we'll likely have to revise uh our statements to a degree. >> Yes. And I understand the state needs that information. >> Yes. >> In a timely way. >> Yes. And as was in our memo, they uh we have been in dialogue with the state. And as

011you can imagine, this is not an issue for Portland public schools alone. Uh currently the requirement is for the state to receive our annual financial report that would include the single audit um information before uh by the end of December so that we could re receive by statute our January state school fund payment. And so there's a procedural challenge there that they're facing and and will address at the state level if the guidance doesn't come out soon. >> So you just have to wait to see. We just have to wait. I think one of the things that we can share with confidence is we are fully engaged with our auditors. They are doing their testing per um all of the controls that we've already done. So we will have a financial report and those financial

012uh reports won't change necessarily. It's only the testing around the single audit that may change. So we may have to go about uh committee review and then even board approval a little bit differently this year. So I would ask for a little bit of flexibility if we need it uh maybe in December. Okay. Thank you. So currently um the plan is to have an audit committee meeting on November 6th. Uh, so we will have another opportunity to get a status update before our December audit committee meeting, but for now it seems unlikely the external auditors will be ready to present the results of the audit in the meeting we have scheduled for December 4th. Right. >> Yeah, they will present uh information. That's up to the committee. So, we can present a draft of what

013they have. It just won't have their final opinion issued at that time. So we can do it as the information comes available to us along our original timeline or we can alter it depending on how many steps we want to take. Okay. As of now um being pretty sure it's not going to be ready by December 4th. Um I've been working with the board office to find an alternative date in December. And as of now, um, uh, I have I have a meeting invite that's been sent to you to hold a spot for the audit committee committee meeting on December 17th at noon instead of the December 4th meeting. I suppose we should keep four as a whole just in case. We really value the input and expertise of our community members and we've tried

014to make it easier to serve on the committee by not starting our audit committee meetings until 5. But for the December 18th meeting to accommodate everybody's busy December schedules, the noon meeting was the best that we could come up with. However, we would like to have the meeting be virtual and we will only have the presentation of the CFR audit results agenda. So hopefully the meeting will be less than an hour. So I guess that's all we can do is prepare for anything. >> Thank you. >> Okay. Thank you, Michelle. Okay. The last thing I want to mention related to this topic is that each year and this is particularly for uh Christie and um Oh, Christiey's she's probably listening um and Stephanie to listen to. It looks like Stephanie is having trouble with her

015um Zoom. Anyway, >> Chair Sullivan, I just want you to know I'm absolutely listening with wrapped attention to Broadway Bridge listening to every one of my wonderful >> I've had to swing through all sorts of traffic to get to the steel and I am now on Williams. So, I'm getting close. >> Okay, so this is >> it's been a it's been a 50minute odyssey from Southwest Portland at this point, but I'm I'm hanging in there. >> I'm sure it's very enjoyable. Okay. Um, so this is really for uh Stephanie and Christie. Um, last year I was one of the board members that met with the auditors uh who request to interview one or two members of the board. When I got the notice from Rosanne that the auditors wanted to meet with me, Janice had

016already informed me that the auditors typically want to meet with a couple of board members each year to complete a risk assessment questionnaire. When I met with them, no one actually called the meeting a risk assessment, and the meeting felt much more like a conversation instead of a checklist of questions. Also, there are no right or wrong answers. They just want to better understand the board's perspective. I just wanted to share that experience with you just in case either of you, Christy or Stephanie, get a request to meet with the auditors this year. Often the auditors do request to meet with one of the board members that sit on the audit committee. Just know that it's just part of their routine audit procedures and there's nothing to worry about. Also, if the request does come

017to you, please try and schedule that meeting with them at your earliest convenience. If you want more information before you meet with them, please just give Janice a phone call and she will be happy to explain why the auditors want to meet with a couple of board members. Okay. Now, moving on to the next agenda item. And this is one um where I'm glad that uh Virginia is there. Appreciate you being here. Um this is a discussion of the external audits of our B bond program. When we asked our community to support our bonds, we promised to be good stewards of those bond funds. One of the ways we demonstrate this is through the accountability and transparency provided to our community via the bond accountability committee or the BAC. Another promise we made to our

018community is to get annual performance audits of our bond program. For the annual bond performance audits, we contract with a firm called Schelberg Evans Consulting, which we refer to as SEC. We have to get another acronym in there just at school district. Uh when SEC has completed the annual performance audit, the audit committee asked the external auditors to present the results of the audit to us. Last year we got the presentation from SEC in June and with the graduations taking place in June, scheduling was extremely difficult. So I asked Janice to reach out to them to get an idea of when they anticipate their report will be ready. This year the report will not be ready to present until June or July. So I would like to plan for July 2026 audit committee meeting for

019the purpose of receiving the annual bond performance audit. This will be tricky because we do not typically have any committee meetings in July and typically the current makeup of the audit committee ends on June 30th. Then typically it is August before the chair of the board assigns the chairs of the committees and then it takes time for the new chair of a committee to get their meeting scheduled. I do not want to wait until August or September to get their report. We have not always been consistent in making sure those audit reports are delivered to the board or one of its committees and it's very important that we follow through on our commitments to our community to provide the oversight of our bond funds we promised. I am going to reach out to the chair

020of the board to see what can be done to figure out a way to arrange a July audit committee meeting and I'd like to invite the facilities committee you Virginia um and the bond accountability committee and that's probably a committee I know a committee that Christie will be involved with and have those three groups join together for the report from the U bond auditor. Ideally, we can extend the current makeup of audit committee members to end at the end of July instead of the end of June. Again, I will need to talk to the chair of the board to see what our options are, but I will be doing that and I will keep you updated through the year. There is one other topic related to the bond performance audits that I would like to

021discuss with the audit committee. Yeah, so strange to see people on the screen. I really want them here. Um, and that is related to the implementation of the recommendations that the bond auditors make in their annual reports. Again, this is something very important for you to hear, Virginia. When we received the presentation from the bond auditors in June of 2025, the year six report included 23 recommendations. In addition to the 23 recommendations in the year six report, they stated that we had an additional 20 recommendations that were made in prior years that the district had not yet implemented. So that's a total of 43 outstanding recommendations. To help me better understand, I asked Janice to go through the outstanding recommendations with me in detail. Oh, yay. Here's here's Christie. For example, one of the recommendations

022that was made in the year three report was the PPS board should revisit revisit its equity in public purchasing and contracting policy to clarify and define the district's vision, goals, and commitment to business equity. This should include reassessing the overall intent, determining how equity is characterized, and expanding on what attainment of business equity goals for the district should entail. And the recommendation goes on to state once the equity in public purchasing and contracting policy has been board affirmed OSM the office of um um >> school modernization I'm so bad with those OSM um should work with other PPS departments involved with developing and implementing the district's business equity directive. And then it goes on to list five additional recommendations that have to do with policy. Basically, what that means is that the board needs to

023revisit its equity and public purchasing and contracting policy before OSM can address the additional five recommendations. For some context, the year three report was issued in November of 2021, and to date, the board has not revisited its equity and public purchasing and contracting policy that really needs to be done so that OSM can utilize that updated policy to address the recommendations. What it appears like now is that OSM is not implementing the recommendations when it's actually we as the board that needs to revisit our policy first. They really can't address that problem until we do. As we walked through the other recommendations, Janice and I started to wonder if the audit committee is the best place to follow up on the status of the implement implementation of other recommendations. For most of the recommendations outside

024the ones for the board, the industry the indust industry knowledge and expertise around the bond program already exists in our bond accountability accountability committee. That's where you're involved and the facilities committee which is Virginia. I am really glad you're checking in on this Virginia as a result. >> I just I was just you're crushing it. As a result, I asked Janice to go through all the recommendations and identify which committees would be best to follow up on the recommendations. And she has done that. She has also met with the chair of each committee to explain why their committee would be best to follow up on the outstanding recommendation and gave them an idea of what following up on those recommendations might entail for their committees. Janice reported back to me that each of the chairs

025of the other committees seemed willing to follow up on the recommendations that she identified for their committee. All of that is great and I think we will we will see some progress on the implementation of the outstanding recommendations this year. The reason I want to talk to you all the audit committee about this is because technically the responsibilities to follow up on the implementation of the recommendations resides in the audit committee's charter and it is not in the other committees. However, I think we have found a way around this. So I wanted to run it past you. In the audit committee's charter, it states, "The audit committee will review all performance audits done by external auditors and make recommendations to the full board and management about reporting, monitoring, and corrective action plans." So, I was

026thinking maybe the audit committee could make a recommendation to the full board that the applicable committees take on the followup of the recommendations that were identified for their committees. Although the chairs of each of the committees seem willing to take it on, I do not want to pass off the audit committee's work to other committees without a reason. We've already mentioned the reason about going to the policy committee, but another reason um see my notes here. There's six, here's seven. Another reason um that um that this work could be done in another committee is because the um office of school modernization staff already attend the facilities committee and the bond accountability committees. If we do the followup in the audit committee that would mean OSM staff would need to regularly attend audit committee meetings plus

027all the the other committees that they attend to get updates. With all that being said, I do not think the audit committee has ever exercised its authority to make recommendations to the full board in this way before. So I do not know for sure exactly how that would work. But with the blessing of you all, the audit committee would like to recommend to the full board that the followup on the ex outstanding bond audit recommendations be done in the identified committees. Then I will touch base with board chair Wong to see what he would suggest. It would still be the responsibility of the audit committee to ensure that the recommendations of the bond audit would be implemented. So we would have to do periodic check-ins with the other committees. >> Patty, I think that's a

028great idea. Say maybe maybe it is like a quarterly, you know, the pol sounds like most of it would be with policy and um and with the facilities committee, but maybe we do like a quarterly update to the audit committee, the chairs of the committees. We could come and give you the information if that's helpful. some sort of routine. >> That would be Yeah, that would be great. We could set that up. Um either you could come to the audit committee or the audit committee could come to you to hear your report. >> I'm happy to come to you. That would make makes sense to come to for me to come to the committee. I can't speak. >> Really cool because >> director to pass. >> I'd be happy to do that. >> Yeah, that

029would be great. and especially since so many of the outstanding uh u recommendations have to do with OSM. Yeah. Okay. Thank you. We'll get that on our agendas. Yes, Chris. >> Uh I would love to hear from the community members first. Defer to you if that all sounds good to you to exercise some strength we haven't in the past to make a recommendation to the full board. Does that feel >> you you have a lot of expertise? >> Yep. >> Yeah, that sounds reasonable. I am not as familiar with these other committees. Could you describe what the makeup of them? >> Yeah, sure. >> I can if that works just to give you a break from talking. >> I know you want. >> Um so as as chair Sullivan mentioned earlier um one of the

030things that's unique about this committee is we have community members on it, right? Um the rest of the committees including the one we keep d um pointing to director Lefort about is the facilities committee. It has a longer name but I don't if it's bioch I don't know it's let's call it facilities. Um that committee um consists of board members and then is staffed by the office of school modernization as well as others in the facilities kind of side of the the shop here at PPS. Meanwhile, um Director Englesman is a member of the policy committee which is pretty much what it sounds like. It's where we review policies, make changes to policies, make new policies. They get developed at that committee and then they go to the board and the members are um I

031didn't say all the members of facilities that's director Chase Miller and director um >> director Deass and then the policy committee is director engass is the chair. Um, so that's the two committees we're talking about that we would ask to tackle some of these recommendations and report back to us under what we're talking about now. >> Great. Thank you. That's helpful. I think that makes sense. Um, as long as we're getting that report back. >> Yeah. We'd have to oversee it, make sure it was done and have people be willing to report to us or we could also go to their meetings. >> Yeah. >> Or one of us could. Yeah. >> Yeah. And it doesn't necessarily need to be a committee chair. Like for example, Director Englesman is on this committee and is also

032on policy. So she might be able to be the liaison if that works for her since she'll be here anyway. Yeah. >> And you're the BAC. >> I'm on the BAC. So I can report back from the BAC um as needed. Yeah. >> Yeah. >> And I'm the le on the BAC. We're talking about the bond accountability committee. I forgot to mention that one. That one's actually no board members are on it. I'm the liaison for it, but it's actually made up of experts in the fields related to the bond work that's being done. Um, and it's an advisory committee for the staff specifically and the board as well. >> Sorry, troops. >> Oh, you're all good. >> And can everybody hear me now? >> Yay, we sure can. >> Hi. Yes. Hi, this is

033Stephanie Englesman. Um, I didn't get to introduce myself early on because for some reason my settings on this phone were not allowing my um were not allowing me to unmute myself. So, uh, I had a child care kurfuffle that was sort of made worse by the fact that there were contractors in the house three weeks longer than they're supposed to be. And I didn't want to leave my kids in the house while the contractors were there. So, I tuned in and then couldn't unmute myself. And now I am driving to the board meeting because the contractors left, the childcare arrived, and my unmute is working. So I should see everybody hopefully in about 10 minutes in person, but I have been listening the whole time. Thank you. >> Oh, I'm glad you're listening. Um, I

034was going to ask for a thumbs up if people think that's a good idea, but it sounds to me like you already have said yes. >> I think so. And I >> keep your thumbs down. The other thing is one of your next steps was to talk to Chair Wong, who's actually here, which is very impressive because he was also up late two nights in a row because I saw him testifying at city council last night after my bedtime. Um, so I don't know if he wants to weigh in and say anything or if not just stay on mute, just listen in and you two can talk later. But if it'd be helpful to just talk now, we could. But he's not coming off mute, so I'm assuming he would like us to not talk

035now. >> Okay. I know he's watching his daughter's soccer game. >> Oh, okay. So there might be a lot going on. >> Fair enough. >> What he's doing. >> Okay. Yeah, I'll I'll talk to him. >> Okay. Thank you. I appreciate that support because it it does seem like >> the best way to get these outstanding things taken care of with people who know what they're doing. Okay. Um, the last thing I want to mention and then we'll get to hear Janice talk um is about our contract with the external bond auditors which expires in the spring of 2026 and we need to do a new request for uh for pro proposals the RFP for these services. Last year when we went through this process for the external auditors who audit our financial statements, a

036couple of audit committee members were part of the panel who reviewed and scored the RFP last year. That was Caroline and myself. Caroline, as an auditor in your professional life, I think you are particularly qualified. Um, and I'm hoping that you'd be willing to serve in that capacity again this year for the external bond. Does that sound good? Okay, I was the other committee audit committee member who participated in the process this year. I was thinking that I would like to offer my spot on this selection panel to the chair of the facilities committee or the board's liaison to the bond accountability committee. That's both of you. So, if either of you would like to do that, um you can let me know later. You don't have to now. And uh that just seems like

037the appropriate people to be on it. Um that would make a selection panel. That would be Caroline. Is it Caroline or Carolyn? >> Caroline. Okay. Thought I could be saying that wrong. uh Janice either um Virginia or uh Christie and the other members of the selection panel will be identified by PPS management. So we don't have any choice of that. This is not going to come up until the spring of 2026. So I just wanted to mention this is part of the work of the audit committee and I will be providing you with more information as we go along. uh director of Fort and I can um fight to the death to see who gets to be on that. >> Get out, Chris. >> Well, be careful. I don't want you injuring each other. >>

038Okay, let's move on to the next agenda item, which is the audit committee work plan and meeting agendas for the 2526 year. I worked with Janice to put this together, but I have done so much talking so far that I am going to have Janice take over. >> Thank you. Okay. In your packet of documents, the audit committee work plan and agenda meetings for the 2526 school year is document number three if you want to take a look at it. Um what I did is I took the primary responsibilities of the audit committee and also the audit committee work plan for the year. And I made sure each element was popped into one of the audit committee meetings that we have scheduled. So for example, one of the primary responsibilities of the audit committee is

039to provide oversight and support of the office of internal performance audit. So what does that actually look like? That is for example um the audit committee and internal performance audit charter needs to be reviewed and and referred for approval if there's any edits necessary. Um the annual performance audit plan um the annual operating budget for the office of internal performance audit and the auditor's annual performance evaluation. So for the charter review, last year we didn't build that into our work plan. So this year I made sure it's on the May 7th meeting date. So, if you flip in your packet to the May 7th date, you will see that um we have a section here to reflect on the audit committee's work for the 2526 school year and review the policy and charter to consider

040any updates. Um, and then another example, if you flip to the January or February meeting date, you will see that we included the discussion and approval of the 2627 budget for the office of internal performance audit. So for each of the elements um that are included in the primary responsibilities and um the work plan I made sure they connected tied directly to an element in one of these meetings and I'm pretty sure I got them all in there. Um and of course that gives us a lot of work to do this year. In addition to putting each responsibility and work plan item in here, I also made notes in here about when potential action would be necessary from the audit committee. Um, so for example, if you are um looking at the well, I already

041created a July one um just in case we ended up needing that, but one of the um potential actions that needs to happen that may need to happen is referral of community members to the full board for appointment to the audit committee. um if we haven't done that already, both Jennifer and Caroline's um they're the on the second year of their two-year term. So although I'm sure we would welcome them back if they you know we we might need to um add another one if anybody else need needs a break. Okay, so that is everything that I put that I put into the work plan and meeting agendas. If you want more details about the primary responsibilities of the audit committee, um on the audit committee's website, there is a whole page that breaks down

042what does that look like and each of the elements. So, just FYI, the website has been updated. Thank you, Cara, for helping me with that. Um, but that's how I came up with with popping all of these in. So, feel free to take a look at it and if you think I missed something or have any questions, you know, send Patty an email and she will tell me what to do. Um, okay. Well, I think that's all I got for that item. Are there any questions? >> So, if we think of things that should be added, we can call you. >> Yes. Call me and I'll call you. >> And also on the agenda for each meeting, I built in a spot for our chair of the committee to ask the committee members if there

043are any requests for audit committee agenda items. um as part of our wrap-up for each meeting. And I built that in just so we don't forget to ask each other that and take that into consideration if there are any suggestions. >> I just have one tiny easy note that I was thinking about under the audit committee work plan on item number three. It says provides ongoing oversight of the implementation of recommendations. And I was wondering if it should say like implementation of recommendations from past audits. It's just like a way to be clear that we're not talking about just I don't know what, but just to make it clear we're saying from and then it includes all past audits. So I know we're working on you know taking a look. We just talked about the

044contracts audit and still needing quite a bit of work there, oversight work there. So, just want to make sure that we capture that it's not just current ones or the most recent one or something like that. >> I'll make an an update to the page. Thank you for that feedback. >> I know. Thank you so much. I love that we know it's coming. It's really nice to see this level of detail. Appreciate it. I when I was mentioning the little paragraph in this book um about community uh members I wanted to mention this whole thing is really good to read if you want to know about what auditors or the audit committee is supposed to do. um because I've I've been on that audit committee now this is my third year and I'm still learning

045so much and there's so much to learn but this was really um well pretty easy to read almost almost fun to read maybe I'm becoming an auditor I don't know my heart >> anyway fun it was good reading so that really helps too in sort of knowing where we're going what we're trying to do okay thank you Janice Um, let's see. Really appreciate and like I think I forget who said that they love knowing what was coming up. I feel the same way. Okay, let's on to move on to the next agenda item and that's the discussion of the OIPA audit plan and discussion on new audit topics which is what I'm looking forward to. Janice, can you go over this? >> Yes. Thank you. Okay. In your packet, this document is the item mark

046number four for you at the top. So I just wanted to start by giving you the current status of the audits that are already approved. Um so the first one is the English language learners and the current status on that is I actually sent the draft report to staff today. Um so I when I was working on putting this document together um I knew it would be this week but it was today. So just FYI um that what what will happen is we send it to the staff first and we give them an opportunity to review the audit report and look to see if they have any feedback for us. And these are the staff that we work directly with. Um, and we also send them a list of every single item and that we

047identified as, you know, a concern and they get an opportunity to review that and make sure we didn't miss anything. Um, and then once that done, we will meet with them and go over those and then move the report on to management, PPS management. So they will have an opportunity to also give us some feedback and um then they'll have a chance to include management's response into the audit report and that is also included into the actual audit report. So when it comes to the audit committee it will include management's response as well. The next audit that we're working on right now is the measure 98 audit. So, we're about twothirds of the way done with our testing here. Um, and so we anticipate this will probably be ready for the audit committee in the

048January or February committee meeting. And if you look at your meeting agendas, you'll see that I did pop that report in there for that agenda in anticipation of that. Um, I also just wanted to mention that the Secretary of State's audit division did a state level audit on measure 98 in this last year. So, I did reach out to their audit team to talk about what they included and more specifically what they did not include in their audit. And so we um kind of tweaked our scope just a little bit to focus on the things that they did not include in their audit. So we're not duplicating any of their work. >> Um the next audit, >> can I ask a question about that one really quickly? Yes. um when you since you're going to

049be kind of relying on the findings from that state level audit, would you be willing to provide like a summary of kind of the high level findings or something within hours so that we can kind of know what those findings were versus just skipping them? >> Would that be reasonable? >> Thank you for that feedback. >> Thank you. Appreciate all the work you have to write it really quick. Okay. Um yes. All right. The next audit that we already have approved is the building security audit. So for this audit, we have done all the background research and identified criteria and created an expectation of generally what would we expect to see out as a security plan out in the buildings. And then what we will do is ask the district for their building security plans

050for each of the buildings that and we're not going to test all of them, but we will test some of them. And then we'll go out on site and look to see that the district is complying with the building security plan that we have already I that the that is already our plan. We'll just be testing to make sure we follow through with the things that we said that we will do um for this audit. We anticipate approximately getting starting started on this um in January or February. And also for this audit, I do plan to reach out to the student representative to the board to try to arrange some student engagement so we can hear from students about how they feel about building security in their schools. So, we will do our very best

051to incorporate that and hear from our students um when it comes to that. And I will work with our student representative to try to make that happen. Um, okay. And if you look at your agenda listing, I've already popped that in for May meeting because that's approximately when we anticipate that we'll be ready. The next audit is the career and technical education program audit. Since this audit was approved, the CTE program has developed and implemented a strategic plan. And I recently met with the director of CTE and Pathways to gain an understanding of their um strategic plan. Um based on kind of an updated understanding of where the CTE program is, we would suggest rescoping the audit. Um, so what that would look like is uh because we were initially planning to look at accessibility

052access to different CTE programs um throughout the district. So like are there schools that don't have access to specific programs? Um well in their strategic plan for CTE it clearly identifies like page four where each program is is at. So I think it's very comprehensive summary of where everything is and it's pretty easy to see which schools do not include programs and which schools do include programs. So I really don't think that would be the best use of our time to do a full audit. However, when I was talking to the director of CTE, um I got some really good ideas in that discussion. So, what I would like to have is an opportunity to follow up with them and try to come up with um a scope of audit that would really add value

053to the work that's already being done. Um so, my plan, if that's okay with you guys, is to bring an updated scope to proposal to the November 6th audit committee, and I'll go over that with you at that time. I still think there's a lot of value in doing a CTE audit. I just want to make sure that we're spending our time on items that um are really going to add value. Does that make sense? >> Yeah. Did Did Chris have some ideas about what would be useful? >> I'm sure he does. >> Okay. Yeah, that would be the >> Yeah. I mean, and he was very receptive to being audited, which was really exciting. He said, you know, he welcomes an audit anytime. So, um, that was very refreshing. Okay. So, that's what we

054already have planned and kind of when you'll expect to see them come across the audit committee this year. So in addition to that, we would like to get some approval on our next set of audits um to add to our audit plan. We need the approval on the audit topics so we can begin gathering the necessary research and criteria over the course of the year so we can start on these audits in the 2627 school year. So for today, what I am hoping for is that we can narrow down the list of the audit topic ideas that I included to maybe three or four. For the three or four topics, we will take those and build out a formal audit plan proposal to bring it back to the audit committee in its next meeting for

055approval. between now and the next meeting. Um, I really need to make sure and circulate the the audit topics um with PPS management team and the leadership team to um make sure and get their feedback on which audits do they think would be val of value and that information and I'll include that in my plan proposal and we will be able to talk about it in our next meeting and take all of that information into account. um and ideally arrive at two audits to add to the audit plan. Um just as a side note, it does require the full board's approval to add audits to the audit plan. So the audit committee ideally on the 6th will refer the selected audits to the full board for approval. So now we can talk a little bit

056about each of these audit topic ideas. What I did is I emailed out to all of the board members including those who are not on the audit committee. Um and I sent it to our student representative and of course our community member representatives on the audit committee and I asked for input. Um, what I sent was a complete list of all of the audit topic ideas that have been suggested or have been talked about over the last few years here in the audit committee and I ask for input on each. So for any of the audit topic ideas that were on that list and no one said, "I'm interested in this one." Those audits are not on this list. So I pulled those out to kind of shorten it down to just the ones that

057received some interest. Um, and I listed in here the level of interest. As a side note, three was the most interest one audit topic got. So when you're kind of looking to see at the level of interest, three was the most. Um so why don't we just kind of start with the teacher absences and subprotocols. I listed there some of the notes of the risks and potentially the why um we would consider doing this type of audit and I included some notes and comments and also um like OIPA's perspective on what this audit would look like. So, for example, um in the teacher absences and subprotocol audit, I just want to point out that um we want we really want to make sure and be careful with the communication and messaging around this audit. I

058could see a risk there that um we just want to make sure that the audit does not we want to make clear that the audit is not an attempt at restrict at restricting legally protected leave but I think there is a way um to do it without making it feel that way. Um, so I put my ideas in there and what I thought we could do. Um, and do you guys want to talk about it individually? >> Does anyone have Does anyone have thoughts on approach? I'm fine just going through them kind of one by one if that works for people. I'm one of the ones who suggested this one. I actually did a terrible job of responding to your email as as Janice knows and now you'll all know because I'm going to be

059like really excited about one I didn't even say anything to her about. But this one is one that I wanted to put on here and it it really is almost nothing to do with the teacher absences part and more to do with like how we respond to them. Um we know we've had a dramatic uptick in teacher absences. Um, it does seem anecdotally to be linked to the new Oregon paid leave law, which is an amazing law that I don't want to be in a position of questioning. It's more like how do we adapt to a world in which uh a career path that is predominantly women, whether we want to see more men in the career or not, it's predominantly women who are often caretakers of both children and aging family members as they

060evolve through their careers. And paid family leave allows you to both take time off to care for sick relatives um as well as care for children. It has a pretty broad definition of family for example. So you have people who are in a caring profession and therefore probably caregivers in other ways in their lives and of course it's affecting us as a district. And so to me, I think it's more about like taking a look at what other places have done to mitigate some of that. Um, for example, I was just at um a conference where we heard from Nashville Public Schools and one of their priorities has been having um somebody who works full-time in their schools. That's sort of the first line of like unexpected absences. So if there's no unexpected absences in

061that building, there's somebody who can be working with small groups, whatever, like they make use of them throughout the day. But then if there are unexpected absences or there's like a long-term absence in a classroom where that person might be the best fit, then they can use that person to kind of be like a a person that can be that long-term sub in the case of a longer term absence. And it just makes a lot of sense to have somebody who knows the community, knows the kids, knows the other staff, right? So what if we looked at other places and how they deal with um having you know subs especially since we have significantly more than we did prior to the implementation of the law. So I just wanted to share why I what I

062put forward why I think it's important and really suggest that if we did do this other than maybe just setting it up as in there's been a a dramatic uptick. Here's what it looks like. um not necessarily trying to prevent people from taking their earned leave, which to me is really problematic. So I would and I think that's what you're putting down. So I just want to make sure it's clear as one of the people who recommended it. It would be more like how do we adjust our work to accept a new reality? >> Yes, I that's one that was important to me too. And I when I was looking at our website um when was it about 2012 or something there was an audit done about that and there were all sorts of recommendations

063which I don't think anything has any were done. Yeah, because but uh >> but there's probably another way to do it and I mean Janice can look at that one and see could we do a better way or a different way or something that would make more sense. >> Yes. So, we did do just a little bit of background research and um I think looking at potentially California school districts, they implemented a leave program similar to paid Oregon leave quite a few years before we did. So, perhaps they might have some strategies that they're utilizing as a tool part of their HR strategy to um help address this. Not sure, but I do think that we could look for that. And I've also heard of incentive programs. So, I would be very interested in doing

064the research on that to see if there's any opportunity to make some recommendations that might help us with our strategy. >> And and is that really an audit or is that just looking at more different ideas or are they the same? It is considered a performance audit and we would follow um generally accepted audit sta gagas hold on generally accepted government auditing standards >> for the the full report. So regardless of the kind of the layout or the context, we're still going to follow the audit standards um to present all of the work. So you'll be able to rely on it knowing that we followed all the standards to arrive at a recommendation if that's what we do. Okay. >> Um I think that um I would suggest that there be a little bit more

065clarification upfront about um what sort of performance improvement recommendation measures we could take. So, what I'm hearing is we are not, of course, going to in any way try to impede into people taking their legal right. Um, I'm not sure and I hope you don't mind me saying this, but like how having another person in the building doesn't reduce the amount of leave that people would take, right? So, are we actually trying to get at mitigating learning loss? And how do we maximize instructional time in light of a new circumstance? >> That's really what we're doing needing to do. I think >> that is great. And you know that's kind of the thought behind tacking on the subprotocols to the kind of the name of the audit. Um because perhaps that might be the avenue

066um that would help mitigate learning loss in that situation. Um but I'll be watching this meeting back to listen to exactly what Jennifer said to write down. >> Yes. >> Thank you. That that means >> Jennifer, I'll just give you an example of something I've noticed and you've been touring even more schools, but I've started touring some schools um and um very often you'll go into a classroom and the principal will see there's a sub there and literally be like, "Oo, okay, well, let's move on." And it's like because they know that the likelihood of like learning or something interesting to see happening is reduced. And I don't and I we have wonderful subs, right? But we also have ones that show up and there might not be sub plans or name all the reasons.

067And so I just think there's I think a lot of us have experienced as parents or going into schools that we just have some work to do to if we're going to have more subs, how do we help those subs be more prepared? Um right like doesn't Yeah. So I guess that's what I'm getting at. But it is it has been sort of almost funny now that I think I've gone to three schools and literally like you see a sub and they're like okay let's go. And it's like okay. >> Yeah. It it seems like the increase in absences is more of the context and the audit would not be of the absences. We just have to be clear of that. That would be on what we're doing to address that subprotocol probably highly in

068there. >> Okay. Do you want to like uh so I'll talk about the next one. Let you know what we have listed here. So support system for principles. This was an idea that came from the secretary of state's audit division. They gave they did an audit of the department, the Oregon Department of Education and PPS and they issued that audit report in 2019. You know that audit report indicated that the board should ensure that the district's strategy plan um addresses challenges in the organizational culture including the division between central administrators, principles and teachers. Um so what this one was the one that got three and um three yeses. So this is the one that seems to be to have a lot of interest here. So I wanted to just kind of mention um the timing

069of this might actually be really good right now. Um you know we're aware that the district has done a variety of work in this area already over the last few years. specifically um in 2021, PPS was one of eight US schools that were selected to for a grant uh the Wallace Grant Wallace Foundation grant and um the purpose of that grant was very specifically to build an equity centered pipeline for hiring and training principles. In part, the Wallace Grant Foundation The Wallace Foundation grant funding allowed PPS to focus on preparing and training principles with tools and strategy to promote racial equity. Um, strategically recruiting and hiring equity centered principal candidates and fostering inclusive environments that will set up principles for success in the long term. Um, so when the district is undergoing work, a multi-year

070pro process to address a problem, it's generally not a good idea to try to go in and do an audit midway. Um, however, the I'm pretty sure that the 2526 school year is year five of the five-year grant from the Wallace Foundation. So if we were approving it now and we wouldn't start on it until next year. So that last year would have been wrapped up and we would be able to take a look at, you know, kind of the results of all of the time and effort that the district has invested over the last few years into addressing this very specifically. Any questions or y'all want to talk about this one? I guess I I guess I'll just give um my like a nod that I feel like this would be good. Um I

071love the idea of kind of celebrating um some success because my guess is that if we looked at sort of the issues we were having with retention of principles that's probably improved over the past several years. So I think it it'll be a good like um pulse check on something that's been a priority that the district has worked hard on. So I think it's a good idea. I love that multiple people are interested in it. >> Yeah, if I totally agree with that and also it would show maybe there's areas they still want to work on because they're I'm sure they know they haven't reached utopia yet, but um yeah, I think that's a really good one. So, I'm going to um I know Director Leffort has come off um video, so I want to

072make sure she can say something, but I'm going to um take an opposite approach and say if we out of these eight or so greatlooking audit suggestions can only pick two or three, I'm not sure that I would want to focus on what is hopefully going to be a feel-good story and maybe just find some things to work harder on or more on because I think this um the teacher absences one is getting some um support. I really like looking at the student performance measures and frankly somehow blending it with the stu the summer expanded program because that are both student performance measures and then I am also really interested in number eight which is looking at the contract and so if there's so many great ones I think the work needs to stay focused

073on finding what we can do to make the district even better. I love the idea of a feelood story, but I'm not sure that I want to. >> I never get to do feel good. >> I know, but if we only You can do all eight, but if we can only do two or three. That's my That's my thought process. >> I was going the same place Stephanie is. I wonder um and I don't really know. You might be following some principles in developing this document. I wonder for each of the points if we're going to try or potential audits if we're going to try to make a decision next time if we can add a section about how they would how the audit and the recommendations would positively impact the outcomes for the board

074goals particularly for the student um performance measures that we if we tied it to working toward the specific goals that are set for the district. >> That's a great idea. And I jotted that down. So when I bring So we're gonna pick three or four today and then I'm gonna do a bunch more work to build out an actual proposal to really identify what will the audit actually what will our objectives be and I will also include um how the audit and potential recommendations would make an impact on student outcomes. So that's great. I'll make sure and include that element in the proposal when it comes back to you guys next time. This document was very high level just to get the conversation going and see what the level of interest is here. But this

075is not a formal audit plan proposal and you will get something much more formal and and well put together when we come back. >> Definitely didn't mean they should have been there. I meant as we make the decision. Yeah. So, >> oh, Virginia, I see your hand up. >> I learned to raise my hand. So, I just wanted to um So, I was laughing when Director Englesman was talking because I was like, they're going to think we're coordinating or something because she basically said exactly what I was thinking. I was looking at student performance measures and thinking about um how some of the other things on this list, would they be captured? um like the summer expanded program offerings, um our personal services contracts to support our students. Would would the results of those um

076programs be captured in the student performance measures audit or could could they be in some way? And I feel like I think um you know when you emailed us asking for our input I have a feeling I think I sent I was just going back through my emails and I was like god I think I sent back my ideas and then didn't actually go into your spreadsheet. So I'm wondering when I see the um you know level like the level is it that only three people responded to >> No no no more people responded. >> Okay. Okay. I was like I know I responded. >> No and I definitely got your your very long email. Thank you very much for that. Even though it wasn't on the form, I certainly got the feedback that you

077were sending my way. >> Um, but yeah, I I really like the student performance measures one on here. I feel like I would love to have one of one of our selections, you know, directly, you know, tied to our board goals. I mean, everything of course is it has a place, but when I see something like this, I'm like, okay, how are we doing? >> Great. Well, why don't we start talking about the student performance measures >> really quick before pardon >> for I wanted to just say something really quick about the principles. Um because normally I would be very much like wanting an audit that's really focused on >> you know d more directly like what is best for students but as a parent I know that like the principal can really have a

078huge impact in a school. So, um I would also put my support behind this one even if it's not as a direct connection. Okay. All right. So, student performance measures and just you know we don't get to pick all of these guys. I am sorry. Um okay. So, for student performance measures, I would really like just to bring some attention to um I'll call her out. Caroline gave me some great feedback on this one. So, um, when we looked and talked about this one specifically, what her suggestion was to break this topic out into a series of audits, one on each of the board um, goals to So, I really like that idea. Sometimes there's this thing called audit scope creep. And it is so incredibly hard to stay focused on just the thing, the

079one thing that you're working on. As you go through the work, you kind of see something else and it's like squirrel. No. Um, but seriously, like it is very difficult. if we were to do a full student performance measure audit that would be inclusive of pretty much all of the goals. I do think it was an excellent idea to suggest breaking them down into a series of audits. So, um, and I really like that idea of a series of audits because there's a little bit of consistency in that work, which would help us be more efficient in turning audits around. So, if we did one on this board goal this year, we would take we would know that we're going to do a series and so the next year we would work on the next

080one or something along that way. um or along that process. Um just kind of my personal thoughts about how potentially this type of audit would really work best. Um, and I really appreciate, by the way, having another auditor on this committee because, you know, like the feedback she gives me is just really spot-on and would really help make sure that we stay focused and not take too long on delivering reports. So, that's kind of my overall thoughts about the student performance measures audit. Do you guys want to share your thoughts? Sounds like we already y'all already shared some thoughts. Um, but what do you think? >> Can I just ask a question? >> Um, so Janice and I were talking about having some um because this the student as originally pulled together. It seems like

081so much like just a huge scope and it also really seems related to what I hope is um a board sort of strategic plan um where we are regularly getting updates that cover some of this content in other ways. So I was a little bit like are we is this something we can do outside of an audit? Um is that what we're aiming for with like a strategic plan? So, I was like, that was kind of my thought on it is like I'd rather this be like the instead of us having an audit that happens a year and a half or two years from now because we already have the audits for this year booked out, something we start doing sooner in a different way. Um, so that was just like my initial thought on

082it, but I do want to share that I I I of course like the idea very much and I think that tying it to the board goals and making it more compact makes sense because we what we were talking about what I started out getting at was like having some audits that are like a little bit more like we're going to look at this thing and it's like pretty um tightly defined would help you have some projects that can be a little faster. to get done. So, um I like the idea of breaking it out if we do. I mean, I'm not going to fall on my sword about thinking we should do it sooner um in a different way so that we can do it sooner. Um but, um but I'm I'd be fine

083with doing an audit down the road with it, too. Um but I love the idea of breaking it up um to make it so that it's um something that could be done in bite-sized pieces. I also wouldn't mind if it became like a series where it wasn't like once a year we're doing one. But because this is so urgently important and if we can't find another way to do it, then maybe it becomes like we do like we do them two or three two or three or even four like as depending on how that bite size works out. That makes sense. >> Okay. I just want to say we I promised we'd be through by 6:30. So we're gonna have to >> Okay, we got to get through. Sorry. Okay. The next one is school

084audits. Um so there are a variety of internal controls that are performed out at the school buildings where um the district office doesn't provide much if any oversight of those things that are taking place there. So for example um leave approval that is all done out at the school. Um, and there are a lot of risks associated with leave, like comparing actual leave to what was approved and and making sure all the leave hours were properly accounted for. Um, and that's all taking place out at the building, which is awesome and great, but it would be a really good idea to take a look at the school-based internal controls in addition to, you know, looking at the district level. There are some that are very siloed out at the schools. Um when we did the

085student body funds audit, one of the things that we noticed is each school seems to have their own way of doing things, which is also okay. But um one of the things we noticed is some of them are have great tools like one school has this very fabulous worksheet and um I would love just to be able to send that to the same you know each school and be like hey look this is a great worksheet and um so I would like to really consider doing a series of school audits where we do a couple of schools each year um and look at the internal controls for things that are just done at the school. Um, another topic that was pretty popular whenever I was at um, Seattle School District was the nutrition and there

086were concerns about um, you know, student access to vending machines and um, not necessarily the most healthiest choices that may not necessarily align with our overall district nutrition plan. So it would that would be one of the things that we could look at out at each school. Um, another item that we could look at is field trip approvals. So I think for the ones that come to the board, they are dialed in. they go through all the review process and um but for the ones that don't have to come to the board, I I just don't know what the visibility is and the oversight procedures for approving and making sure they're properly chaperoned, etc. So, that would be something that we could look at out at the schools. Additionally, um student body funds. So when

087we did the student body funds audit, we did um provide some recommendations and also one of the things that we noticed um that we made note of when we talked about the implementation of the recommendations of the student body funds audit. We talked about this last year and um one of the things we talked about is that they don't have staffing for the o for them to go out and do their own school site mini audits. Um that was something that was done in the past and an FTE was cut from their department. And so I really think there would be value in perhaps the internal audit department kind of picking up a little bit of that slack by including a check-in on student body funds as one element of a schoolbased school sitebased audit.

088So those are my thoughts. Oh sorry one more thing. A lot of school district internal auditors have a mandate to do school-based audits on a regular basis. So that was something at in Seattle school district that was a mandate. They wanted us to do at least one school-based audit on internal controls that are at the schools every year. Um, we don't, that doesn't have to be the case, but that is the case sometimes. And we don't really have a mandate. Sorry, my we don't really have a a mandate for our internal audit department that says we expect this, this, and this. um which is also okay because it allows us the freedom to you know come up with audit topic ideas based off risk and feedback and input where it's not prescribed that we have

089to do the same thing every year. So there's value in both, but I just wanted to mention that a lot of internal audit departments do have a mandate. Like for example, some of the school districts in Florida, I went to a conference last year in Florida and I got to meet a bunch of internal auditors at different Florida based um school districts and they have a mandate to do building security audits every year. So that is just part of their routine. Every year they do X number. Of course their internal audit departments are significantly bigger than than what we have. So of course we need to keep that in mind which is also okay. But I just wanted to give you an example of something that you know sometimes school audits are mandated every year

090and sometimes like a security audit. But um we don't have that which is okay. But just wanted to throw that out there so you would kind of have an idea of how comparable we are with other internal audit departments. >> And so you do like three schools one year, three schools the next year. Okay. Okay. Well, let's move on to the next one. So, this is the results of the 2021 summer expanded programming offerings. And just to be very clear, we're really not interested in going back to 2021 specifically, but an audit of our summer program offerings more broadly. um sounds like a really good idea to see if um what the results are of the summer programming and whether or not it makes sense to invest more um potentially in summer programming depending on

091how the outcomes look. But that is um kind of the thought behind this um topic. the when this was requested initially, it was in 2022. Um so that's why the the 2021 was listed, but um you know perhaps looking at summer programming, you know, over the course of a few years might be a good idea to see what the outcomes are compared um comparatively. But I did put in here what the audit that was suggested. But I wanted just to clarify that we're really not looking to look at 21 specifically, but a little bit more broadly about the impact of summer programming. >> So h how do you actually do that? you take the kids that went to summer school compared to kids that didn't go to summer school in the same class that >>

092I'm not exactly sure but I will build out if this is a what you guys want I'll build out a proposal on how we would go about and I'll have to do some more research and talk to people about what data is available um and look to see how we could come up with an audit that would add tell you. Okay. The next one is the incident response procedures and communication. So when students come to me and ask for an audit, I really do my best to try to at least bring it forward and and mention it to see um if there's interest in doing it. So, in this situation, um, two students who stated that they were aware of what they considered a significant security threat on campus and they were not provided any

093communication from the school about what happened or what was going to happen to make sure it didn't happen again. Um, and they were wondering, you know, does the district have a formal policy and administrative directive about what communications is expected to students when there is an incident out at the schools. Um, so this did not get a lot of interest, which is okay, but um, I definitely wanted to mention it and bring it forward because it it is my practice to try to bring forward audits that are requested from students. It doesn't happen very often. So, when they do actually come to me and say, "We'd like an audit of this," I I do my best to be responsive to that. Um, one thing that I wanted to mention is that during the current year,

094we are going to be looking at the building security audit and during that audit, we do plan to do some engagement with students and that might be an opportunity to hear more about this potential communication concern that they may have. So from from my perspective, I think it might be a good idea for this one to let us do the building security audit and then kind of gauge with the student um engagement what the little just learn a little bit more about the protocols and stuff like that and then potentially consider it in a future year. Okay, next one is the complaints process. So, um, the complaints process policy was updated in 2023. Um, and one of the things that I've noticed is that I don't see complaints coming to the board as often anymore

095than than we did um, a few years ago. it really wasn't uncommon to see on the board agenda or a special meeting to um take into consideration a complaints. So maybe that's because we have no complaints which would be awesome, right? But it also could be representative of something else. So um you know the complaints data is of value to you board members. Um when I was chatting about this one specifically with Patty, she did let me know that you guys get emails um sometimes saying there's no complaints or or something. I'm just not privy to that communication. So, I'm not sure exactly how it works or um but it does potentially sound like there is some communication flow. So, that makes me feel better because not seeing it consistently and then having it kind

096of go away did make me question like what's going on. Um and maybe we're just doing it in a different way which would be okay but it >> something might >> it might not I mean I don't know. So that's why the it's listed. Um, okay. Then the last one is a contracts and contract management audit. And um, you know, I did get a a really strong recommendation to consider this this type of audit. And um you know, I really see that and I hear the communication that you guys have in your board members uh or in your board meetings. I listen and I take notes on things that you're talking about and it and I do see your interest in our contracts here at PPS. So, I wanted to present it because it was

097suggested, but I also want to say that we did do a contracts audit back in 2020 and we issued those and we still have quite a few recommendations that are outstanding. So, it would be my preference to try to work on the implementation of those before we started in on the topic again. but it has been five years. So, um maybe that's what you guys want. So, I'm just throwing it out there. So, that is all of the ones that were suggested >> and you need to narrow it down to like three. >> Three. The max is four. >> And that's then you'll work on it. Take it to to staff as well to look at to see if it's >> And then when we come it back, you're only going to get to pick

098two >> in November. Yes. Um, so if we can do four. So can should everybody just think of four they like and we'll look. Does that sound reasonable? >> So let's take a minute to >> Yeah. Yeah, >> Director Sullivan, I do want to add to as I'm going through these and I've been working hard as um Genesis staff contact to make sure that we're a little bit more in sync because I'm aware of other um audits and reviews that are happening throughout the district consistently whether it's federal programs, nutrition services, student transportation, like there there are audits and reviews, assessments happening all the time on top of these things that um we are asking Janice to do. So I want to help provide some visibility to you for those. So just to add assurance

099and not have um duplication of effort if we already have a third party coming in and doing reviews on what's already required by statute. >> Great. Can you do that with these? I I can for some I can I can connect with some uh reports that are already being um compiled. For example, um any uh summer programming, we're tracking that data and and some of it may already be coming back to the board in a more general format like just a staff report. So I just want to share that with you that we want to be in concert so that for the limited resources we have on performance auditing that they are truly on those areas that aren't being thoroughly reviewed already by another party. >> Is there any are there others besides the summer

100one that are on this list that I mean I know the contracts is we're doing contracts. Yeah, we're doing the contract ones, but we do have a certain level of testing that happens as part of our annual comprehensive report. And so that that comes to you ultimately as well, and they would note findings uh if we were out of compliance with with policy. So I I just want to bring that to you guys so that you have an additional layer of assurance and so that our internal performance auditors can be really targeted. Um you know, I think one of the challenges has been to bring a report and get in uh the recommendations implemented because they've been a little bit broad and I just want to make sure that you guys know there are other

101things happening and we're starting to gather a repository of information to share with you. And I will add to that that just this last week or two when when I was having my regular check-in meetings with Michelle, she was talking about that and I got so excited because this is um one of the things I've been trying, you know, I would really appreciate knowing about like there are a lot of like OD will do like a desk audit or desk review of some programs or whatever and you know I just don't have any visibility into that currently. But it sounds like Michelle is going to start gathering that um and is willing to share that. So that would be very helpful for me to um come and suggest and propose audits that would be, you

102know, more meaningful and potentially of more value. Um, one of the things that I plan to talk about a little bit later, but time is ticking, is about a formal risk assessment. So, one of the procedures that, you know, we're really required to do by audit standards is a formal risk assessment. And currently to date, we have really been relying on the risk assessment that was done back in 2016. Also, I've really been leaning into the Secretary of State's audit report to help identify risks here. Um, so it it really is time for us to actually put together our own risk assessment and that will be a lot of work. Um, but one of the ways that that will by what what she said would be so helpful is because I would be able to

103gain an understanding of all the different types of things going on in the different departments. that would help me know what risks are mitigated based off of some of the other types of procedures that are being done by other auditors or maybe not auditors but other program evaluators necessarily or um something like that. So when we chatted about that, I I was very excited about that and the timing is really good because it is my goal for the next time I come to you guys to talk about audit proposal topics that I'll have done a risk assessment to have a little bit more of a basis for why am I suggesting these audits. Um, did you want to say? >> Yeah, I have two things to say. One, I have to I have to leave,

104but I've signed it back on already to be present again. Sorry. Uh, we've had some childcare issues as I said earlier for this number eight. While I find it very interesting and I've circled it and started, I also wonder whether it's not better suited as a board agenda or work session report to understand how and I understand we had co but if there was this contracts audit done uh five and a half years ago and there are 20 recommendations 11 of which have not been implemented to just sort of understand what obstacles there may be to not implementing those final 11 or you know because that feels more like a work session or board meeting agenda item rather than another audit to me. >> I think I think we're like I think I think you're

105on it because Dr. Adams was supposed to be here tonight rescheduled instead for October 20 board meeting to talk about just this. >> Okay. So Oh, this is all wrapped up in his report that was >> potentially. So it is >> I would like that to be wrapped >> Yeah. >> Not on mic. I would like that to be wrapped up in that report. I do anticipate the report that he's working on will be the evidence that we're looking for to demonstrate the district has implemented some of the recommendations. But there are going I mean I do not anticipate that it would cover all of the outstanding recommendations, but I do think it was going to cover two of the big ones. So that would be a great win. But um one of the things

106that I'll get to really quickly because I I know we have to go. But um in my annual report, I am asking the audit committee for some additional support in relationship to the implementation of the contracts audit recommendations. Specifically, it has been our practice to ask the um management to come and give us updates on the status of the implementation of recommendations annually. But our charter does allow for us to do that quarterly. So I'm asking for the audit committee to request that quarterly and in anticipation that you guys would be super supportive of that. I already popped that into our meeting agendas. So you will see a lot you know we will have um you know management come to talk about these pretty consistently this year. Um, and I'm fingers crossed. I really anticipate

107this year we're going to get these dialed in. >> Back on Zoom. I'm so sorry everybody. >> Well, and I pro I promised you we'd be through at 6:30 and I had no idea it would go so long. So, good discussion. >> Present the whole time in person. >> Okay. Um, well, you need to get four. >> Yep. I need four, please. >> So, I we could talk about it forever, I think. So, I I just really would like everybody to choose four and then we'll vote and see which four gets the most. I I can't think of a better way. Okay. Unless somebody can. >> Okay. >> Each one, see how many votes it gets >> and then that's it. Yeah. >> Okay. >> Take a minute. >> Teacher, let's start with teacher absences

108and subprotocols. >> Okay. Wait, wait, wait. I'm still getting my top four. Okay. >> Okay. We get four votes down. Yeah, we get four. >> Okay. So, okay. >> You get to vote on four. Is that what we're doing? Okay. >> Yes. Want me to make sure you've got four chosen because I'm not sure I do. Okay. All right. Okay. Number one, teacher absences. Um, I guess raise your hand, I guess. Or show. Okay. One, two, three, four. Um Stephanie, she raised her hand. >> Okay, >> nice work. Good job, director. >> But I I just want to we again suggest we might entirely reframe this as support for subs. >> Yes. >> Okay. >> Yes. Thank you. >> Yes. >> Yeah. >> Okay. So, we got five votes for that one. What about the

109support system for principles? >> Okay. About Yeah. Raise your hand for number two. support system. One, two, four. Are there four, >> Stephanie? No. Okay. All right. The student performance measures via the series of audits. Okay. Four plus Stephanie, so that's five. Okay. School audits. Any interest? Yes. Two. >> Two. Okay. um results of summer programming. Okay. The incident >> director too for that one as well. >> Oh, okay. Thank you. Sorry, I missed her. >> Um the incident response proced. >> Actually, I put my hand down. I meant to I put it up and then I changed my mind because we only get three or four of these. So, do not count me in that one, please. >> Okay. I'm I'm removing you from that one. Um okay. incident response procedures and communication. >>

110Okay. Um >> can I just I should have said this earlier. I I meant to chime in and say there's two of them. Uh this one might be one of them that I wonder if we refer to the policy committee of which I'm on because if we don't have a policy about how things are reported to students and I imagine it can be very worrisome to students when they know something is going on and they also know that information is perhaps being kept from them. This feels like um perhaps start with the policy committee and someone tell me if I'm just missing something here, but maybe that's where this goes. And if we have a policy, we look at it and does it still make sense? And if we don't have a policy, we look

111at that and think whether we need one. >> Oh, that's a great idea. I'll look it up to see what our policy is. And um if we do or if we don't, I will follow up with an email on this one to the chair of our policy committee um based off of whatever I come up with. So even though um maybe there's not a lot of interest in doing a maybe it might be something the audit committee could pick up >> or not and I think that policy committee. >> Oh, I'll say I'll speak up when we get to the one that I think this also applies to. Okay, next one for thoughts is the complaints process one. All right, last one is the contract management one. Okay, >> even the person that sent it

112to me is not voting. That's okay. Um, okay. So, I think I hear what you're saying. So we definitely have interest in the um potential support for subprotocols um support system for principles student performance measures and then we have a hand up for director Englesman. >> Yeah, thanks. I realized that we must have skipped the other one that I wanted to bring up as potentially belonging in the policy committee and um it is the I think it's under the school audits one which I couldn't see how many hands went up but um the board policy for outofstate travel is on the Monday agenda for potentially being changed and if there isn't board policy on how schools IND individual schools work on both accepting leave requests and which the leave requests might also be looped into

113the sub one and the first one like those seem very connected but if there isn't a board policy currently on uh just local travel and how that's approved perhaps that belongs in the policy committee as well maybe we don't need a policy on it but it could be something that the policy committee could take on similar to the other one I uh could end up in the policy committee. >> Okay. I'll reach out to you also next week to kind of um hear a little bit more about suggestions for the policy committee um because I think you've got some great ideas and I would love to hear a little bit more. So I'll reach out to you next week >> for a little bit more details about which you think maybe could go to the

114policy consider for consideration. So okay well I think that we have three strong support for the first three the teacher absences and subprotocols the support system for principles and student performance measures. Then the kind of support drops off a little bit for the remaining ones. So, um, do you guys want to pick one more for me to build out or just go with the three and just know you only get to pick two of the three? >> I just have a quick question for Michelle. Did you just say we're already looking at the summer learning stuff or did I mishar you? >> No, I know for the Oregon grant, which I don't know that PPS was a recipient of, but >> not this year, but we were in past years. Yeah, I know that we

115keep all of that student data and so it's more of a program request to say how did our summer program go? How are those students doing currently? I feel like those those are valid questions for our instructional leaders that would they would probably love the chance to share that information. >> Anyway, I just I thought I just wanted to clarify because I think that might inform the rest of the conversation. >> Will you say one once more what the top three were? >> So, one, two, three. Okay. Victory. >> Yeah. Okay. I think I heard the direction. So, I will prepare. I will do a bunch of research and come back with a proposal and then you'll get to pick two of those. >> Do we I wonder how close the student performance one I

116wonder how close was that in interest or was it very low? that just seems so linked to board goals and district goals and everything that I I wonder why we're what again what I'm missing on that one that um >> how close was it? >> Student performance measures got a five. >> Yeah. So, it was one of >> Oh, that's one of the three that we're talking about, >> right? >> Yes. >> Oh, I'm sorry. I don't have that obviously in front of me and I missed that name. Perfect. Glad I'm glad it's there. Thank you. >> It was most popular. >> Yeah. Okay. So, that wraps up our conversation in relationship to this topic. Um, we are out of time. >> I know. So, can I just get two minutes just to say a

117little something here for the overview of the office of internal performance audit? Um, and the attached policy and charter essential conditions. Um, you know, I have met individually with each of you guys to go over this information in detail. And um you know for the newer board members, specifically Stephanie and Christy, I sat down with you guys this summer and we did a full hour of talking about internal auditing here. So I have included all of the documentation and if you guys have any questions about any of the topics, please just give me a call and we'll talk about it. If you're wondering what I would like to point out as something that's super important, that would be the list of essential conditions. And in that list of essential conditions, there are board identified essential

118conditions. So, if you have time to spend a little extra time reading this, focus in on the board essential conditions and please feel free to reach out to me if you want to talk about any of them in specific. >> Okay. And Janice, you also can you very very briefly um go over the audit report web pages? >> The audit report is document number or not the annual report is document number 10 in your packet. >> Um so I I kind of buried the lead earlier where I mentioned that I was asking for additional support from the audit committee in relationship to this. So in the audit in the annual audit report, if you flip to um page four of six, this is a section that I want to talk about because last year I

119didn't include this section in this report because at that time I didn't feel like we had any significant outstanding recommendations. Um but this year we do. So I am just drawing attention to some of the outstanding recommendations in the contracts audit and asking for some additional support by getting quarterly reporting on that and I preemptively popped that into the agendas already. Um another thing that we didn't have in here last year that we have in here this year is the audity feedback survey. So, thank you to uh Director Sullivan for taking the time to reach out to all of the staff that we worked with for all of the previous audits. So, she did all of them for the last five years. So, we got up to date and she summarized that information and provided

120it to me. One of the issues we had last year about doing this type of survey is we couldn't figure out a way for it to remain anonymous. So Patty was able to, you know, and that's important. We want staff to be able to give like their direct feedback and without fear of upsetting anybody. I don't know. But like the being anonymous is important. So thank you so much for doing that. I know it was a lot of work and I really appreciate it and we kind of listed the feedback in here and you can also see the list of all the questions that are asked. Um >> um Stephanie has her hand up. >> Sorry. Go ahead De. >> Yes. Hi. Sorry. This is um unrelated to what Janice was just saying and I

121appreciate that and certainly the um anonymity. I wonder whether and this is something that's been interested interest I found interesting. I wonder if in the student performance one is you build it out if there's a way to look at and this came up a few times with people talking to me as I was campaigning about like what would you what would you cut if you're not cutting student facing things and you're like things that the district is paying for. teachers were bringing to my attention, and I've seen it with my own kids, how much time they put into getting ready for and then sitting for all of the various testing mechanisms. And I believe that we use three different ones, certainly um maths and OSAs, and I think there's a I think there's a third

122one. And so I wonder if in building it out because it's one thing to get all of these performance results from various testing data but if we are taking up a lot of otherwise important instructional time that could be used um whether it's worth it and I I don't know whether that's something that could be identified or quantified but it's certainly something that is interesting to me and if our kids are sitting for days upon days upon days of different testing throughout throughout the school year that's already a short school year. Should they be? So, I'll just leave it at that and maybe someone can tell me what they think. >> That sounds like a whole another audit. >> It does. >> That's not on the list. >> But Stephanie, I will reach out to

123you to dig into that idea a little bit more. Um because it's very sounds very interesting to me. Um, one thing that we at internal audit does is when the schools ask for support, um, I oftent times let Mary Katherine go and be the support person. So, for example, she went out to I think it was Lincoln just yesterday where and served as some >> Yeah. like a pro, I don't know, not necessarily a proctor, but was there in the room to support students taking their tests or whatever, >> proctor? Yeah. >> Um, and so that meant that she had to take the training also. And even though you know we just did that to be supportive of that, but we also learned so much about the process by taking the training and it helped

124build out that kind of thing helps us build out a more rounded understanding of all of the different types of processes and procedures. But since you mentioned the testing, I just thought I'd throw that in there to just let you know like we do things like that to help build our understanding of different processes and procedures in addition to the actual audit work that we do. >> Okay. So, you know what you're going to work on. Yes, we did that. And thank you for quickly going over the rest. You're welcome. It's hard to do it. Uh I'm amazed we took this much time. Okay. Um, the wrap-up item, I just want to confirm our next audit committee meeting is November 6th. It's listed at 5:00 PM, but if we could do it at 4:30, uh,

125is that okay? Because uh, all I think all the board members have been invited to Sunnyside for a big talk about funding. So, I think it's important we go to that if we can. And we have a much smaller agenda that day. So I think we'll be out. >> Well, we'll we'll try. Although I may go on longer than we think. >> Okay. >> Um let's see if there's anything else. And just after I said we tried to meet the you what your needs. We asked if you can be earlier, but thank you. I appreciate it. Um I guess that's it. Oh, you wanted to ask if people are okay with all the paper or if they'd rather have it. >> So, for me, I'm you can tell I'm an auditor. I tick and tie

126things, but it's helpful for me just to facilitate the conversation to not have to pull it up on my screen and share it. So, I was just wondering if there's any preference like do you does it does it help or do you prefer to have it pulled up on your screen? Um, I'm just looking for some feedback so I can be more efficient. >> Well, anybody who spent five minutes with me knows that I print everything out, so I appreciate it. >> Okay. >> Um, but that doesn't mean that if we go virtual, I'll probably just still print it out. >> Okay. >> Yeah, I'm good with the paper. Thank you. >> Okay. >> I'm fine either way. >> Okay. I I like paper. I can find it better. >> Me, too. I mean, I

127lose it, but then I >> Do you have a preference? >> Yeah. I want to be clear. I don't want paper for our actual board meetings, but I love the paper for this one because it's like a good amount. Although, I would like printed agendas. I keep asking for it and it's never happened, but for Thanks. >> Okay. >> Thank you so much everybody and thanks for staying late. And Stephanie, thanks for running back and forth and making it work. >> Thank you. Well, thank you everybody for being so with your grace for my crazy um afternoon. It was nice to see everybody. Good night. >> Good night. Thanks. Oh, the meeting is adjourned. >> Thank you.