| ||
 |
Once agreements have been established, authorizers conduct varying levels of ongoing oversight relative to those agreements. Substantial philosophical and capacity differences underlie this variation. Some authorizers argue that over-regulation will stifle charter schools - that too much government involvement will lead to the recreation of the bureaucratic system from which charter schools were meant to escape. They believe that if a school stays out of the papers, keeps officials informed about potential problems, and submits its reports on time, then it should largely be left alone. However, other authorizers disagree. They believe that regular and ongoing oversight will enable the authorizers to determine whether a charter should be renewed or revoked and perhaps might even help schools improve.
Capacity is another major consideration in properly analyzing data. Some authorizers, such as Arizona's State Board for Charter Schools, suggest that even though they have an interest in visiting all of the schools they sponsor, they simply don't have the capacity to do so on a regular basis. As a result, the schools are largely kept in check by parents who may leave if a school is not addressing their child's needs, or who may alert authorizers when schools are out of compliance with a law or regulation. This section summarizes how various authorizers gather data to assess schools' progress toward the goals in their accountability agreements. Since the kinds of goals included in accountability agreements differ from authorizer to authorizer - and, in some places, from school to school - each authorizer's approach to data-gathering will be unique. Different types of goals create the need for different types of information, and authorizers design their data-gathering systems accordingly. Generally speaking, though, authorizers rely on two kinds of information:
Reporting Requirements All authorizers use some form of reporting requirements. Some authorizers rely on these requirements as their primary data-gathering tool, while others use them to supplement school self-reports and on-site review processes. Five different kinds of "paper" reporting that authorizers may choose to review include:
Most charter schools are required, often by law, to report their progress on an annual basis. These annual reports, or school report cards, generally require that the school report its progress toward the goals established in its accountability agreement. The column to the right gives examples of two formats for reporting. Compliance Reports Many authorizers also require charter schools to submit other reports more frequently than the annual report. While compliance reporting is not the focus of this report, it is an important and unavoidable part of running a charter school. Often, these reports are procedural requirements imposed by law or regulation. For example, they may have to submit data about student attendance to substantiate their enrollment numbers that underlie their per-pupil funding. Some data in these reports may be of little value to authorizers while some may be useful as authorizers consider a school's progress toward its goals. Minimizing the required work Taken together, reporting requirements can be overwhelming for many charter schools. Before imposing additional requirements beyond those legally mandated, authorizers should think carefully about three questions.
The central component of any authorizer's review of reported information is likely to be an analysis of the results of student assessments taken by a school's students. This may or may not be a part of the annual report. Required data In nearly every state with a charter law, charter schools are required to administer certain standardized tests to their students - perhaps official state tests, perhaps a common national standardized test such as the Stanford-9 or the ITBS. According to the National Study of Charter Schools, more than 96% of charter schools administer standardized tests to their students.18 Additional data: standardized or alternative? Beyond these requirements, though, charter authorizers need to decide what, if any, additional assessments they will insist upon, and what, if any, additional assessments they will accept as evidence of student achievement. A general principle suggested by some authorizers is that the assessments used to measure performance should be aligned with the school's goals. But that simple principle masks a complex set of controversies about the value of different kinds of assessment data. There is a long-standing divide over the best way to measure student learning.19 Those primarily interested in large-scale, comparable assessments depend on on-demand assessments, commonly known as standardized tests, because, when designed and used properly, they are reliable and consistent from one administration to the next, and valid (actually test what is being assessed). Others argue that students learn, and thus convey their learning, in ways that are more varied than an on-demand assessment can measure. They call for a more diverse set of assessments, including those that allow students to demonstrate knowledge and skills through means other than scores on multiple-choice tests. These alternative forms of assessment go by names such as "performance assessment," "alternative assessment," or "authentic assessment." For example, City on a Hill Charter School in Boston, Massachusetts, calls on over 100 professionals from the community to conduct "juried assessments" of student work. These high school students are asked to demonstrate their competence in the core subjects - for example, justify in layman's terms how they solved an algebra problem, or explain a passage from "The Iliad." While these assessments provide students with a broad array of means to demonstrate their competence, they may not be consistent from student to student because jurors are not trained. Some charter school authorizers are attempting to incorporate data from alternative assessments into their evaluations. However, schools and authorizers using alternative forms of assessment face a number of challenges:
As a result, on-demand assessments tend to emerge as the predominant mode of measurement unless school leaders and staff can devote an extraordinary amount of time to creating, testing, and refining performance-based assessment systems. Practitioners often have more faith in performance-based assessments as measures of their students' learning. However, central bureaucracies including many charter-authorizing agencies, rarely have the capacity to go out to schools and review actual student work, or even to judge in the abstract whether a school's system of assessment passes a test of rigor. As performance-based assessments become more common or popular, third-party evaluators may emerge to examine schools' systems and hand out "seals of approval." In the meantime, authorizers are on their own. All measures of performance are imperfect, so many authorizers attempt to gather data from multiple vantage points in order to gain the most accurate and comprehensive picture of a school's performance. Parent Survey Data Authorizers may be interested in knowing how parents, teachers, and perhaps students judge the school's performance. Data about enrollment and waiting lists provide some information, but some authorizers have taken the additional step of surveying families periodically about their schools to gauge satisfaction. For example, the DC Public Charter School Board's list of common indicators for all charter schools includes a series of Likert-scale questions each school must pose to its parents, teachers, and students in surveys. School Self-studies While the annual report gives schools the opportunity to present a wide discussion of their results and progress, some authorizers have begun to ask schools to take the additional step of conducting more in-depth "self-studies." These studies, similar to those conducted by schools as part of accreditation processes, engage a broad range of school stakeholders in an effort to examine the school's strengths and challenges. The findings provide the authorizer with useful information in reviewing a school's performance and can also prove useful to the school itself if school leaders take the self-study seriously as a tool for improvement. Self-study #1: an example The DC Public Charter School Board and the Colorado League of Charter Schools have developed two of the most detailed self-study processes. In DC, schools chartered by the DC Public Charter School Board undergo a self-study in the spring of their first year of operation. Schools examine their standards and curricula, their instructional programs, their approaches to assessment, their school and classroom climates, their management and governance, and parent and community involvement. On each topic, schools complete a form describing their proposed program, reviewing their current status, and summarizing problems needing attention and recommended actions. Schools compile these reports and then present them to external teams conducting site visits eventually resulting in an "action plan" negotiated with the authorizer to remedy any shortcomings identified during the self-study or the site visit. Self-study #2: an example The Colorado League of Charter Schools' plan calls for two rounds of self-study - one in the school's second year and one in their fourth. The second-year study paves the way for a low-stakes visit by an external team in the school's third year. The self-study's findings form the basis upon which the school's progress is judged in coming years. The fourth-year self-study provides the basis for the school's high-stakes renewal application. In both cases, schools use a set of "critical questions" and receive technical assistance in the self-study process from the Colorado League of Charter Schools. See http://www.charterfriends.org/accountability/cfi-accountability8.html# for "critical questions." Self-study's dual purpose causes tension Tensions can emerge when self-studies are intended to serve purposes that are both formative - helping the school improve - and summative - helping the authorizer make decisions about renewal and revocation. If school leaders believe that data from their self-studies will be used by authorizers to arrive at summative judgments, they are likely to try to "put their best foot forward." In other words, even if visitors are meant to be "critical friends," they are like to be seen as more critical than friendly. Therefore, schools will have some incentive to downplay or even ignore significant areas of challenge. But a tendency to sweep difficulties under the rug undermines the formative value of the self-study. Schools that treat the self-study as a promotional effort will miss the opportunity to use it as an opportunity to reflect upon their practices and make plans to improve. There is no easy way to manage this tension. The Colorado League of Charter Schools' plan places the self-study early in the school's life, significantly before the renewal stage. The results of the self-study do not feed directly into decision-making about the school's renewal. Instead, areas of challenge identified in the self-study become some of the bases on which the school's progress is judged over the rest of its charter term. During the renewal process, the authorizer asks, "How effectively has this school addressed the challenges it identified in its self-study?" Though this approach does not completely eliminate the tension, it helps elicit honest responses from schools to the self-study's questions. Benefits Of Reporting Requirements Reporting requirements can be a valuable tool to inform the public about their options and serve as guideposts of performance to the authorizer. For example, they can be used to flag problematic schools and help determine whether a school's charter should be renewed. An accountability system that is solely reliant on reporting requirements may also be beneficial to schools because it is relatively unobtrusive. And from an authorizer's perspective, reporting requirements are beneficial because they are inexpensive and can be imposed with minimal staff - a real advantage if authorizing staff capacity is low. Drawbacks To Reporting Requirements While there are many advantages to reporting requirements, exclusive reliance on this oversight method has shortcomings. The first is that much of the data are self-reported and thus not completely reliable. One senior policymaker in Massachusetts, commenting on the dearth of good data from the traditional public school system, said, "we have no idea what actually goes on in the schools across this state." Officials rarely verify the information from schools unless it deviates dramatically from past years' reports. However, unless these data are verified, the schools have no incentive to take their reporting obligations seriously. Second, paper reports provide only a limited perspective on a school. If the school's goals go beyond objectives easily measured by standardized tests and other readily available information, a great deal may be gained from walking a school's halls. On-site Reviews Several authorizers (Chicago, Massa-chusetts, and Washington, DC) have begun conducting site visits to complement the data gathered via reports. The plan developed by the Colorado League of Charter Schools also includes several site visits by external teams. In some cases, site visits were inspired by the British inspectorate model and the U.S. accreditation process. These visits open up a range of design issues including:
Site visits can be informal or formal in nature. In an informal visit, an authorizer staff member visits the schools to "check-in" with the directors or follow-up on a complaint. In Michigan, Central Michigan University staff sit in on the charter school board meetings and regularly visit all of their charter schools at least once per year. These visits generally involve at least a conversation with the school director, and perhaps other observations and discussions. For many authorizers, parent complaints drive the informal site visit process. For example, if a parent complains, the authorizer sends a staff person to the school to follow-up. Whether these informal visits are scheduled or not, they can help the authorizer gain a sense of the general well-being of the school and perhaps address a problem before it becomes more serious. In the case of formal visits, authorizers often send individuals or teams to visit the charter schools in an area. Some district superintendents in Georgia and California send one staff person per year to each school. In contrast, the Massachusetts Board of Education conducts a mix of individual and team visits, beginning with a pre-opening check-in before the school begins operation, and concluding with a multi-day visit in the last year of a school's charter. The DC Public Charter School Board's accountability system and the Colorado League of Charter Schools' accountability plan include similar types of visits. As the accountability cycle boxes reveal, one critical issue surrounding formal site visits is their timing. The Colorado cycle calls for site visits every two years, while Massa-chusetts requires three site visits in the same five-year time period. Staffing On-site Reviews The size and composition of site-visiting teams vary. In Colorado, some district superintendents hire retired school administrators to visit each charter school. In Massachusetts, on-site reviews generally involve a team convened by the authorizer that includes a member of the authorizing staff, a parent, a teacher, a school administrator, and perhaps, a local business representative. For its renewal inspections, Massachusetts contracts with a company, SchoolWorks, that trains and contracts with a range of education professionals to serve on inspection teams. The Colorado League of Charter Schools' review process, which is used or under consideration by several Colorado districts, uses charter school peers on the teams. Though these approaches are different from one another, they all share one common characteristic: a reliance on individuals outside the agency to conduct on-site reviews. Using outsiders serves several purposes. First, it addresses the capacity issue that constrains many authorizers. Most charter-granting agencies simply cannot afford to employ a sufficiently wide range of staff to provide all the expertise required to conduct effective site visits. Second, outsourcing provides a level of political distance that many authorizers seek. The report of the site visit team comes not from agency staff, but from an outside group of experts without the history and politics of the authorizer itself. The composition of teams is crucial to the credibility of this approach to monitoring. While some charter leaders are concerned that reviewers can be overly harsh, some reviewers may be viewed as charter advocates who are willing to overlook a school's poor performance and inadequacies. For example, a Massachusetts Board of Education member questioned the rigor of the Massachusetts process because its site-visiting teams were too "charter-friendly," in his estimation.20 It's likely impossible to find a team that all parties can be happy with, but the best advice seems to come from the Massachusetts Charter School Inspection Handbook which states: The most effective inspection team represents a range of backgrounds and experiences and an unwavering commitment to consider a school on its own terms.21 How to prepare site visitors Authorizers must also consider how to prepare site visitors for their roles to ensure quality, consistency, and integrity in all the reviews. At least three different approaches have emerged from existing accountability systems. One approach is to hire a private contractor to train reviewers and conduct reviews. SchoolWorks, the company hired by the Massachusetts Department of Education to conduct its renewal inspections, provides intensive training to all members of its team and equips each with a detailed Inspection Handbook. Under a second approach, a charter school association conducts the training. For Colorado schools using its accountability plan, the Colorado League of Charter Schools trains team leaders and provides them with a Team Leader Booklet outlining a process they can use to orient and prepare members for their work. Finally, the DC Public Charter School Board is an example of training provided by the authorizing agency itself. Processes And Protocols The protocols that authorizers follow vary depending on the goals of the visit. In Massachusetts, for example, teams engaged in annual site visits generally meet with representatives from all of the school's constituent groups - parents, students, board members and administration - as well as review student work during the course of one day. Consistency of visits On renewal inspections, the Massachusetts teams spend three to four days at the school, exploring its workings in much more depth; team members, for example, may shadow a student for a day. Authorizers are only beginning to develop detailed protocols for these visits, and many still conduct them informally. As an authorizer expands the number of schools in its "portfolio," however, it becomes increasingly important to ensure that the types and quality of the data generated by site visits are consistent across schools, review teams and time periods. Consistency requires the creation of clear protocols and for training site visit teams in the methods they are to use in their observations and reporting. Duration and frequency of visits The duration and frequency of visits vary as well. Since they may be high-stakes visits, they have to be long enough and comprehensive enough to capture a fair perception of the school. However, since many site visitors may be volunteers and the availability of authorizing staff is often low, scheduling a comprehensive one-day visit per school is a significant commitment. For example, to coordinate, conduct, and write-up the 30 one-day site visits that took place in Massachusetts last year required one full-time staff member and half of the office manager's time. The multi-day renewal visits were coordinated by outside consultants at a cost of more than $10,000 per visit. In designing site visit protocols, authorizers will need to assess the costs and benefits associated with the level of oversight they intend to employ. Site visits of longer duration and higher frequency yield more data and much valuable information, but they do place a heavier burden on the schools and the authorizer. Preparing for a site visit Another issue worth considering is how authorizers ask schools to prepare for site visits. In the processes developed by the DC Public School Board and the Colorado League of Charter Schools, site visits are not passive events. Charter schools are expected to complete a detailed self-study before the site visitors arrive. Through the self-study, the school identifies its strengths and weaknesses prior to the visiting team's arrival. This process:
Finally, authorizers need to determine the content and form of the report that site visitors will produce after completing the visit. Since the form of the report should follow the questions that the visit is intended to answer, it will vary from place to place. For examples of report formats see the Massachusetts Charter School Inspection Handbook and the Colorado League of Charter Schools' Site Visit Team Leader Guide (http://www.uscharterschools.org/pub/uscs_docs/r/co_team_ldr_guide.htm). Targeting Of Site Visits Which schools should be visited? As stated previously, site visits can be expensive. For some authorizers - especially those with large numbers of schools across a vast geographic area - the cost of site visits may be prohibitive. As a result, some might argue that authorizers could save money by reserving site visits for only clearly troubled schools. But some authorizers disagree. Massachusetts officials, for example, argue that visiting all schools is advantageous because:
The benefits of site visits appear to be at least threefold. First, they can serve to provide a much fuller profile of each school. Site visitors can review student work, enriching the picture presented by the school's standardized test scores, and they can visit classrooms and gain an understanding of the ethos of the school. Michigan authorizers reported that simply "getting a feel of the school" by walking its halls was very informative.22 In essence, site visits can both confirm and augment by providing a window into school successes and shortcomings that might be obscured in paper reports. Second, site reviews can serve to catch problems early. For example, YouthBuild Charter School in Boston, Massachusetts returned its charter under threat of revocation in its third year. It had already stopped receiving funding in its second year, based on data initially gathered on a site visit. The school was clearly in trouble. It had far fewer students than promised, it was organizationally in chaos (e.g., four directors in two years), and it was awarding high school diplomas based on an eighth-grade curriculum. Authorizing officials believe that if they had relied solely on the school's own self-reported data, they may not have caught and corrected this problem as quickly. The third, and perhaps most important, benefit of site visits is that they can serve as a lever for improving school performance. One charter school leader in Massachusetts said: We're like islands. We're so wrapped up in the day-to-day we don't often have a chance to get honest feedback. I welcome it. We may have done these things on our own [aligned all of their grade-level benchmarks with the state standards and looked more critically at test score data] but the state site visit certainly pushed us along.23 Another Massachusetts charter school principal suggested that site visit reports could serve as a necessary lever: After our second principal left, the site visit report indicated that the board needed to back off and stop micro-managing operations. Everyone in the school knew this had to be done, but the board wouldn't have listened to us without the added weight of the site visit report.24 In sum, external site visits appear to have the potential not only to monitor, but actually improve, school functioning. The Shortcomings Of Site Visits Despite their benefits, site visits have shortcomings as well. They are costly. If conducted by authorizing staff, site visits require a great deal of staff time to coordinate, participate in, and document. If conducted by an outside consultant, they are expensive. Further, site visits are only as good as the visitors and the training the visitors receive. Because taking part in a site visit is time-consuming, unpaid team members will likely only participate in one visit per year, and their exposure to training is unpredictable. Therefore, while the data gathered have the potential to be extremely beneficial, the expertise and training of the teams may vary from visit to visit. This inconsistency can be reduced by, for example, compensating reviewers, but its presence demands that site visits not be the sole measure of a school's success. The Pros And Cons Of Making Data Public As information is gathered on each charter school, authorizers face the questions of how much of the information should be shared and how to share it. In other words, how transparent should the process be?25 As discussed previously under "annual report options," the most readily available electronic or hard-copy data are school report cards or annual reports. In a market-based system, customers need good information in order to make informed decisions. Unfortunately, annual performance data are often not complete or readily available. Senator John Huppenthal, chair of the Arizona Senate's Education Committee acknowledged this problem, "To make that purchase decision something that means something, [parents] have to have maximum information, almost all of the data that we have nowÉ is [sic] worthless."26 If the goal is to raise student performance, what information should be made available to parents, schools and the general public to achieve this end? Public information is a double-edged sword. Its release is necessary in a public system - especially one based on choice. However, the release of some information can be damaging rather than helpful. For example, a hard-hitting site visit report could serve as a wake-up call to the school, but it could also be used by the media to publicly embarrass the school. Laws like the Freedom of Information Act generally restrict the withholding of any information generated by public dollars, but some authorizers have developed strategies that provide adequate information, while minimizing damage to the schools. For example, Massachusetts' policymakers do not release the final draft of a school's charter renewal inspection report until after the Board has voted on its renewal. Other authorizers suggest that an authorizer should not be thinking about the effects of bad public relations on a school, as long as the information released is objective and fair. In addition to parents, charter schools can also benefit from a transparent accountability system. Unless the process is clear, school leaders may not understand how the information gathered on their performance will be used. One charter school leader in Massachusetts said, "Even after I got the site visit report, I still didn't know where we stood. The report critiqued some things and complimented us on others. So, are we doing better than average or are we on the verge of revocation?"27 The Colorado League of Charter Schools' approach underlines this point by stating that there should be "no surprises," and that the schools should know where they stand in the eyes of the authorizer at all times. Authorizers also benefit from a transparent accountability system. Providing clear and adequate notice of problems is a prerequisite to actually implementing a high-stakes consequence such as revocation. Thus, in order to create a transparent policy, an authorizer not only needs to publicize its general policies in advance of actions and clearly document its actions, it also has to telegraph its intended actions so that the schools have an adequate amount of time to respond accordingly. In sum, gathering data on the performance of a charter school is complex and filled with many questions. As authorizers embark on this process, they need to assess their own capacity to gather information, determine what they need to examine, and decide on the most beneficial means of using and sharing this information. |
|
|
INTRODUCTION | SETTING THE TERMS | GATHERING DATA | USING DATA | CONCLUSION | APPENDICES AND ENDNOTES
|
||