| ||
 |
In order for schools to be held accountable, charter school leaders need to know to whom, for what, and how they will be held accountable.5 In theory, all of these expectations are spelled out in an accountability agreement accepted by both the charter authorizer and the charter school board of directors.6 This agreement, then, is at the heart of charter school accountability.
Though school districts have engaged in limited forms of performance-based contracting over the years, charter school laws' insistence on placing entire schools on clearly written performance agreements is new in its depth and sophistication.7 District schools have had school improvement plans for years, but because they have seldom had any teeth they have rarely served to push or guide significant change. In the charter school arena, however, these agreements carry considerable weight. If a school does not meet its contractual obligations, it may lose its charter. The consequences attached to the agreement make both the authorizer and the charter school take it seriously. Most charter school laws give authorizers very little guidance about the process of crafting accountability agreements. Though some laws specify what elements authorizers must include in agreements, (e.g., mandatory participation in the state's testing system), they typically say little about the set of critical design choices that authorizers face. Critical design choices include:
The Agreement Form There are several ways to go about crafting accountability agreements. Four options (described with examples to the right) are:
For authorizers designing their own accountability agreements, options 2 through 4 present a range of possibilities. Options 3 and 4 are very similar -- whether the accountability agreement is an amendment to some broader agreement or a separate contract has little substantive significance; authorizers' preferences here depend largely on the legal framework within which they function. The central design issue, then, hinges on whether to rely on the charter school application as the main input to the accountability agreement, or whether to go beyond the application in significant ways. In option 2, authorizers relying heavily on the charter school application ask each applicant to spell out in clear terms the goals it intends to meet as a school over the course of its charter term. The clarity, measurability, and ambitiousness of these goals are among the chief criteria used to judge the application. If the authorizer approves the application, it is simultaneously approving these goals as the basis for the school's accountability. To the extent that negotiations take place between the authorizer and school, they take place within the application process. For example, many authorizers make final approval contingent on certain changes to the charter application - such as refinement of the applicant's proposed goals. The advantage of using the application process as the basis for the accountability agreement is that it gives authorizers confidence that all of the schools they approve have clear, measurable, and reasonably ambitious performance goals from the outset. But this approach also has drawbacks. First, charter school operators typically develop their applications prior to selecting school leadership, hiring a faculty, and enrolling a population of students. They may have general ideas about what they want the school to achieve, but it is very difficult to develop realistic plans in a vacuum. Second, the often-hurried charter application process rarely allows operators the time they need to develop or select a full set of academic standards; establish a range of goals; and select appropriate assessments. Third, this process can marginalize the role of the authorizer, and hence the public, in setting performance standards. The authorizer is put in a position of reacting to the applicant's accountability desires. Accordingly, some authorizers have decided on option 3 or 4 and begun using the first year of a charter school's operation to develop the school's full accountability plan. Though the application may serve as an initial framework, the first year provides an opportunity to refine the plan and tailor it to the realities of the school's actual student body. One advantage of this approach is that it helps secure the buy-in of actual school leaders, faculty, staff, parents, and students to the accountability plan. The process of having an inclusive discussion about the organization's objectives is valuable in itself. One principal in Georgia described it this way: "So many charter schools fell apart because the principal wrote the charter and tried to implement it. Maintaining and sustaining the vision is the most difficult part of being a charter school. There needs to be an internalization stage."8 Developed by the actual school community, the accountability plan can serve as a compass as schools navigate their start-up years. In places like Chicago where the authorizer and each school negotiate a plan during the school's first year, the fact that the final plan is literally signed by all parties ensures that it is acceptable to all. How Much Assistance To Give Applicants Authorizers also face the question of how, if at all, to provide assistance to applicants or schools as they craft their plans. Developing a set of clear, well-aligned standards, goals, and measures is a complex business, and charter school operators often require expert help in the process. Following are examples of how authorizers have provided technical assistance to charter schools. These authorizers all recognized the importance of outside assistance, but they approached the task in different ways. 1st Example: One Coordinating Organization Chicago Public Schools enlisted a local charter school resource center, Leadership for Quality Education, to help create an assistance program called the Chicago Charter Schools Standards and Assessment Project. From privately raised funds, charter schools received competitive grants, ranging from $5,000 to $13,500, which could be used for staff research and expert assistance in developing unique school standards and assessments exceeding the requirements of state law to align their performance contracts more fully with their school missions. 2nd Example: An Assistance Provider For Every School Massachusetts used part of its federal charter school grant to provide each school with $10,000 to hire a facilitator in 1996. The Charter School Office provided a suggested list of providers, but schools could choose their own facilitators to help them develop their accountability plans. Once initial drafts were submitted, the state hired a contractor to create a common template and to help all schools complete their plans. 3rd Example: Working With Accreditation Agencies Charter schools in Colorado, Massachusetts and Arizona are looking to respected accreditation agencies for added validation of their efforts. In Massachusetts and Arizona some charter schools are looking to nationally recognized accreditors - New England and North Central Accrediting Associations, respectively. In addition, new charter school leagues in Arizona and Colorado are attempting to develop their own accrediting processes. See http://www.apba-az.org for information about the new Association for Performance-based Accreditation. Laurie Gardner's "Navigating the Standards Maze," and "One Hand Tied: Defining and Measuring Charter School Performance" give general guidance on standard-setting in charter schools. See http://www.cacharterschools.org for Laurie Gardner's "Navigating the Standards Maze," and "One Hand Tied: Defining and Measuring Charter School Performance" (published by California's state charter school resource center). For more information on developing accountability plans see "A Comprehensive, Practical Guide to Holding Charter Schools Accountable" by Laurie Gardner and Eric Premack (http://www.cacharterschools.org), "Accountability: The Key to Charter Renewal" by Bruno Manno (http://www.edreform.com), the Accountability Initiative of the Charter Friends National Network (http://www.charterfriends.org/cfi-accountability.html), and the "Charter-Granting Agencies Accountability Toolkit," Edition 4 (http://www.cacharterschools.org). Agreement Elements An accountability agreement has two sides. One side outlines the school's obligations to the authorizer - the goals it must meet; the measures it will use; and the processes it will follow for collecting, reporting, and using information. The other side outlines the way the authorizer will hold the school accountable - the methods it will use to gather information and the responses it will deploy based on the evidence it obtains. While authorizers may structure their accountability agreements differently, they still contain common elements, the most important of which are:
These basic elements of the school's side of the accountability agreement can be seen in the DC Public Charter School Board accountability plan template. The essential elements of this plan include:
Setting Reasonable Expectations In establishing an accountability agreement, it is helpful to think about two levels of expectations that apply to a charter school. The first level is a set of standards that the school expects its students to attain. These standards articulate what the school's students should know and be able to do at given points in time. The second level is a set of goals for school-wide performance. These goals relate to how well the authorizer expects the school as a whole to perform - for example, the percentage of students attaining a certain standard, or the gain in average student performance over time. In setting reasonable expectations for a school's performance, authorizers must attend to both kinds of expectations. Are the standards to which a school will hold its students and the goals the school sets for itself reasonably ambitious? For charter schools, establishing reasonable expectations is a difficult task. If school leaders set their sights too low, then not only will authorizers likely reject the agreement, but the faculty may actually lower their expectations. However, if the school sets its goals too high, it could set itself up for failure, especially if the goals are set before the school knows its actual student population. Setting Goals Though both student standards and school-wide goals are critical elements of accountability, this discussion focuses on the setting of goals for the school; whether or not the school meets its goals is the critical issue from the authorizer's perspective. In theory, only if a school meets its goals will the authorizer renew the charter at the end of five years or prior to that if the school falls far short of its goals. Optimally, then, the authorizer and the school should agree upon both the threshold for renewal and the floor for revocation. In other words: How good is good enough and how bad is too bad? Authorizers and schools need to agree upon goals that are clear, measurable, and reasonably ambitious. Making these determinations involves a set of complex design decisions. There are many different types of goals schools can pursue, and authorizers are faced with deciding how to integrate these into coherent accountability plans. Some of the challenges of doing so include:
Balancing Academic And Non-academic Goals Manno suggests that there are two types of goals that should be included in an accountability agreement: those that are academic, and those that are not.9 Extending this analysis, some authorizers divide the accountability pie three ways:
The Colorado League's "Critical Questions" document is on the Charter Friends National Network site at http://www.uscharterschools.org/pub/uscs_docs/r/dc_expectations.htm Another framework comes from Charter Schools in Action: Renewing Public Education which outlines a system of Generally Accepted Accountability Principles for Education (GAAPE), analogous to the finance world's Generally Accepted Accounting Principles. At the school level, the principles call upon schools to disclose information about educational achievement, fiscal soundness, organizational viability, and compliance with the law.10 But how important a role should academic and non-academic goals play in authorizers' decisions about renewal and revocation? In Massachusetts' policy, academic performance is deemed more important than non-academic performance; how much more important is open to interpretation. In Chicago, expected minimums on both academic and non-academic performance are stipulated in the charter school agreement, but again, their relative weight is largely unspecified. It may be that creating policy in preparation for every eventuality is impossible and that any decision will require some degree of professional judgment. However, to the extent possible, it would seem prudent for authorizers to acknowledge that all goals are not equal, and to be as clear as possible regarding their relative weights. An example of setting academic and non-academic goals Somerville Charter School's 1998-99 annual report in Massachusetts, gives us examples of academic and non-academic goals. Their stated academic goal: All students will receive a well-rounded education that emphasizes mastery in English, mathematics, and Spanish. The evidence that they were meeting this goal: on average, students gained 17 months of grade level equivalency over a 12-month period on the California Test of Basic Skills. Their stated non-academic goal: Students will uphold high standards of conduct to create a disciplined atmosphere of learning and will promote ethical, moral and civic values. The evidence that they were meeting this goal: Approximately 1,500 "gold slips" were given for exemplary behavior, double the number that were given the year before. Also, 221 students were awarded the annual certificate for Model Student Behavior. Their stated non-academic organizational goal: to continue to be a viable organization. The evidence that they were meeting this goal: Their budget was balanced and enrollment exceeded expansion projections from the previous year. Balancing Common And School-specific Goals Authorizers face a second question regarding the appropriate mix of goals in the charter agreement: What is the appropriate mix of common goals - those that apply to all schools - and school-specific goals - those that are unique to each school? Charter statutes may answer these questions to some extent, but one can imagine a continuum of responses to this question. At one end, an authorizer could decide that all schools should sign essentially the same accountability agreement. Standards for student performance, school goals, and measures would be uniform from school to school. This approach is evident in North Carolina, where all charter schools are following state-mandated standards, aiming to achieve statewide goals for school performance and improvement, and employing official state tests. At the other end of the continuum, an authorizer could agree upon a unique agreement with each school, with no required common elements. In between the two poles, an authorizer could require that agreements contain a core of uniform elements, but also allow agreements to contain school-specific goals. For example, all schools chartered by the Chicago Public Schools are judged in part on the basis of state-mandated tests and the Iowa Test of Basic Skills which is used district-wide, but they also have the option of proposing additional measures by which they would like to be evaluated. The uniform approach suffers from one of the classic problems bedeviling public school accountability systems. That is, if the contract is too generic it will be less helpful to the school because it is not specific to that school's mission; and it will be ineffective as a motivational tool because the school faculty will likely not buy into the process. Some observers of accountability systems use the concept "internal accountability" to describe the systems and practices schools use within their own walls to monitor quality and use information to improve instruction.11 To the extent that authorizers can align formal, external accountability systems with internal approaches accepted by schools, they are more likely to have success at using accountability systems to affect school performance. At the same time, authorizers may well want to have some common measures upon which to compare charter schools to one another, and to other schools across the district, state or nation. Making those common elements part, but not all, of the agreement helps ensure that schools' accountability plans reflect their unique mission about teaching and learning. The chart to the right shows how authorizers in Massachusetts and Chicago combine the common with the school-specific elements in their accountability agreements. The other chart to the right, below, "Benchmarks Required of DC Charter Schools" lists the "common indicators" insisted upon by the DC Public Charter School Board - beyond these common variables, schools are free to negotiate more school-specific measures with the Board. Weighing Levels vs. Growth In Performance Regardless of an authorizer's perspective, the common challenge is making a determination regarding a school's "academic success." But does success entail meeting some absolute threshold of performance (a certain percentage of students performing at grade level), achieving a certain level of improvement over time, or some combination of absolute performance and improvement? Absolute scores is one approach followed by the Arizona State Board of Education. The Board applies an average cut score of the 35th percentile on the Stanford-9 - a test required by all students in the state. If a charter school scores below that mark, it receives a letter requesting an explanation of its performance. While this approach is relatively inexpensive and may identify under-performing schools, it does not account for the school's "value-added." A school may open its doors to a population scoring at the Stanford-9's third percentile, raise that level to the 28th percentile, and still find itself in trouble. As a result, this approach may discourage serving at-risk youth. On the other hand, a school that started the school year with scores well above the 35th percentile but experienced a five-point drop would be considered an adequate performer. By contrast, some authorizers seek to measure the school's "value-added" by tracking the performance of students over time. For example, an authorizer may measure the gains of students from the fall to the spring in a given year or track a cohort of students from one year to the next. While this approach overcomes the problem of focusing solely on absolute scores, it still suffers from challenges of its own. Unless great care is taken, for instance, it may not take student transience into account, often up to 30% in urban areas. To address this problem, authorizers can ask schools to report data that includes only those students for whom they can report results over the entire relevant period - for example, those students who sat for three successive annual administrations of a standardized test. In addition, an exclusive focus on improvement allows schools to be deemed "successful" when many students have failed to master basic skills. Some people argue that focusing only on improvement institutionalizes low absolute expectations for students who have started out behind. Consequently, many authorizers look at both absolute scores and gains over time. The Chicago accountability matrix illustrated on page 12, for example, evaluates schools on both the percentage of students that meet or exceed national norms and the amount of growth students achieve from one year to the next. Beyond test scores, while the discussion around charter school accountability often centers on results, the process may also play an important role in renewal decisions. For example, if site visits reveal that test scores were poor in years one and two and that the school took corrective actions (e.g., reallocated resources for tutoring) to address the problem, some authorizers may regard this action to be as important as the scores in the last year of the school's charter. In essence, the authorizer would consider a school's trajectory as well as its standing at the time of review. But taking such considerations into account can prove tricky. How can an authorizer determine whether a school's planned corrective action is likely to succeed? Comparing Charter And District Schools Whether the yardstick is absolute scores or gains over time, an additional question is whether an authorizer should rate charter schools by comparing their performance to that of other schools. The chart on page 11 illustrates that authorizers have many choices in this regard. Charter schools will inevitably be compared to schools in their surrounding districts. The media, policymakers, parents, and the general public are likely to examine charter school results relative to one another and against the backdrop of district outcomes. Still, authorizers must decide whether and how to incorporate relative measures in their formal systems of accountability. There is some debate in the charter school world about the importance of relative measures. Some argue that unless charter schools can outperform regular public schools, there is no way to justify their existence. As a member of the Massachusetts State Board of Education told U.S. Department of Education researchers, "Of course charter schools need to outperform their district counterparts. If a kid can get the same level of education in his neighborhood school, how can we justify the expense and bother of a charter school?"12 Others suggest that if charter schools provide options for parents and meet their own goals, the question of their relative success is not so important. As former California state senator Gary Hart, a key sponsor of that state's charter law, said, "Is it the right and a fair question to ask charter schools to provide Ôcompelling evidence that they provide superior education'? I don't think so. Did we pledge they would be better? No. What we asserted is that we would provide meaningful alternative choices for parents and students based on articulated outcomes."13 A number of considerations might influence an authorizer's decision about whether to include comparative indicators. One is any statutory requirement to use comparative measures in decisions about charter renewal and revocation. But since most charter laws are silent on the issue, most authorizers will have to consider other factors, such as the degree to which the broader political environment appears to demand comparative analysis of charter schools; the extent to which charter schools are serving comparable populations of students to surrounding districts; and the feasibility of obtaining meaningful comparative information. For authorizers that choose to include comparative indicators, there are many different ways to approach the issue. Chicago appears to be the only authorizer to address this question in its accountability agreements, which contain the following provision: This agreement establishes the performance levels [listed on page 12] which generate High, Middle, and Low ratings for each indicator. However, the Board shall consider classifying a pupil performance indicator as "High" if the charter school's performance, without attaining the level specified for "High" performance by this agreement, nonetheless greatly exceeds the performance of other comparable public schools.14Methods of comparison There are different methods of conducting comparative analysis. The predicted scores or predicted gains approach statistically controls for demographic factors and develops a projected score or gain for each student.15 If a school's students generally exceed their predicted scores or gains, the authorizer can have some confidence that the school is "adding value." A similar approach involves identifying one or more schools that can serve as a comparison group for each charter school, based on the similarity of their populations. For example, when it came time to decide whether or not to renew its district charters, Los Angeles Unified School District decided to measure the success of its charter schools relative to their district counterparts. Contracted researchers from WestEd and the University of Southern California had the charter schools identify district schools that they thought were most similar to them in size and demographic make-up; the researchers then compared the schools' performance. For more information go to http://www.wested.org/policy/pubs/full_text/lausd.htm One innovative approach to generating comparative data comes from the nonprofit, Just for the Kids, which maintains a website that allows Texas schools to see how their results compare to other schools with similar populations (http://www.just4kids.org). Pitfalls of comparison methods Approaches that attempt to "control" for various factors that may statistically predict student performance (e.g., race, gender, and socio-economic status), such as the WestEd study or "predicted gains" approaches, are not problem-free. First, since schools with high proportions of disadvantaged students may be held to a lower standard than schools serving their more advantaged peers, such an approach has the potential to institutionalize lower expectations for poor and minority students. Second, the statistical analysis involved in these studies is difficult to explain to the public. Third, there are inherent statistical challenges in comparing typically small charter school samples with larger district samples. A small number of high- or low-performing students in a small school can drastically change that school's average. In statistical terms, the "sampling error" is large in small schools.16 Finally, these approaches are vulnerable to questions concerning the true comparability of the samples. One of the biggest distinctions between charter school parents and traditional public school parents is that the former are often "active choosers." Charter students and their parents may be more motivated than their district school counterparts even if they are similar in observable demographic characteristics. As a result, charter schools may outperform their peers for reasons that have little to do with actions of the schools themselves. On the other hand, if families whose children have special learning challenges are disproportionately attracted to charter schools, charters could appear to under-perform their counterparts. A national study underway by Caroline Hoxby, a Harvard University economist, attempts to correct for this problem. She is tracking the performance of rejected charter school applicants with those who are accepted. Since charter schools must take all comers, those that are rejected are statistically equivalent to those who were accepted - they simply were not picked in the lottery. Rating their performance against that of rejected students should provide a more valid comparison of the "value added" by charter and regular schools. Though a study of this type may be unrealistic for most authorizers, those with access to ongoing data about students who do not succeed in charter lotteries may be in a position to conduct such analyses. Analyzing Sub-groups Of Students To the extent that all of the approaches above look primarily at average levels of performance or improvement, they all potentially mask the difficulties of lower-performing subgroups or individual students. Unless the data are disaggregated and incentives are built in to encourage schools to improve the performance of all their students, the school's average may be rising, while their lowest-performing students continue to lag behind or are left out of the picture all together.17 Unless specifically prohibited in policy from doing so, charter schools could follow the lead of some districts, and exempt certain special-needs students from testing in order to raise their average scores. This is an issue of both policy and capacity. As mentioned, policy can be created that will limit a school's ability to positively skew its results. However, to gather, analyze, and synthesize these data may take a level of expertise and staffing that doesn't exist at either the school or authorizer levels. How Definitive Should Performance Goals Be? No matter what types of goals make their way into an accountability contract, authorizers face a decision about how definitively to set thresholds of performance. There are two broad responses to this question: the contextual and structured approaches. The contextual approach Massachusetts authorizers have chosen the contextual approach. Officials have deliberately decided against creating a published matrix of expected performance because they fear it might oversimplify their decisions about renewal and revocation. Massachusetts policymakers believe that charter renewal is a complex decision based on professional judgment. Similar to the way trial juries apply the legal standard of "beyond a reasonable doubt," the Massachusetts authorizers attempt to make their best judgment based on a rich body of evidence. They use a rigorous framework, discussed in more detail later in this report, and employ a range of highly trained professionals to apply the framework. But they stress the importance of professional judgment in the final analysis. This approach has some outstanding features, but it has shortcomings as well. While accountability plans are fairly specific, the state's three accountability criteria are less so. This model generates a great deal of data, but because it lacks clear definitions of success, it is open to political interpretation and influence. Authorizers run the risk of not being able to enforce a revocation decision. And finally, charter school leaders do not know definitively where they stand throughout the process. The structured approach The authorizers in Chicago take a more structured approach. In an effort to make the process as transparent as possible, they have created matrices of expected performance based on a range of measures (see chart below). This matrix is helpful in clearly stating the expectations up front, but it does have limitations. The matrix provides compliance "minimums," but it does not necessarily encourage excellence. In addition, it is conceivable that schools will fall short of specific thresholds and still merit renewal by virtue of special circumstances. Ultimately, specifying performance thresholds on a variety of measures does not eliminate the need for judgment. What if a school crosses some thresholds, but not others? What if a school crosses thresholds in some years but not others? The matrix gives those decisions structure, but does not wholly eliminate the need for professional judgment. The great strength of the matrix approach is the way it invites a systematic process for making such judgments. Setting the terms of the agreement between the authorizer and the charter school is the foundation of the accountability relationship. These terms - established through the application and/or accountability agreement - set the course for the school and establish the expectations of the authorizer. |
|
|
INTRODUCTION | SETTING THE TERMS | GATHERING DATA | USING DATA | CONCLUSION | APPENDICES AND ENDNOTES
|
||