Before presenting the substance of the Accountability Plan itself, we think it will help to offer a more detailed rationale for this effort and some additional thoughts on accountability in Colorado charter schools. Then we will present the Critical Questions that serve as a guide for the charter school during its first few years. Following that, we present the Accountability Plan itself, with specific suggestions for each year.

A. Rationale

The Colorado League of Charter Schools has developed this Accountability Plan in order:

• To provide a plan that will assure the state that charter schools and their districts have a model they can use or adapt that is fair and that fosters a high degree of accountability.
  
  
• To provide a plan that will help charter schools be more accountable.
  
  
• To promote stronger quality control among the Colorado charter schools. To that end, to provide a series of steps that foster school improvement.
  
  
• To provide a plan that will help districts have the information they need for the evaluation, renewal, and accreditation of charter schools. The plan would be particularly useful for districts and charter schools at two key points:
  
  FIRST, when setting the terms in the initial contract, so that the contract spells out (perhaps in an addendum to the contract) the steps, the criteria, and the timeline for accountability that the district and the school will commit to during the first five years of the school.
  
  SECOND, during the school’s fifth year to help with the charter renewal process.
  
  In agreeing to these steps along the way to renewal, charter schools and their districts will have a common understanding from day one of what will be expected, year by year, in terms of the activities and reporting—both the process and the criteria—that will be used to maintain accountability. This provides a much-needed degree of predictability and consistency to the renewal process, and enables both schools and districts to focus on educational issues—not political ones—in determining whether a school is worthy of renewal.
  
  
• To base site visits and external studies on an approach or practice highly regarded among both private and public schools, both abroad and in the United States: creating teams of visitors who spend two days or more observing the school, who then provide a written report, an objective outside evaluation of a school’s progress.

B. General thoughts on accountability in Colorado charter schools

The charter school law in Colorado expects these semi-autonomous schools to answer to the public regarding their progress. The legislative declaration states that the law is enacted "to hold charter schools accountable for meeting state board and school district content standards" (22-30.5-102). Among other requirements, charter schools, according to the law, "shall be accountable to the school district’s local board of education for purposes of ensuring compliance with applicable laws and charter provisions and the requirement of section 15 of article IX of the state constitution (pertaining to school districts and boards of education). A charter school shall be subject to all federal and state laws and constitutional provisions prohibiting discrimination…" (22-30.5-104).

And while charter schools, like other public schools, are expected to provide Annual Reports and School Improvement Plans to their districts, they have an added challenge or responsibility. The law says that a charter may be revoked or not renewed by the local board of education for several reasons, including failure "to meet or make reasonable progress toward achievement of the content standards or pupil performance standards identified in the charter application" (22-30.5-110). Furthermore, in their charter applications charter schools are, by law, expected to describe the "educational program, pupil performance standards, and curriculum…" The application must also describe the plan "for evaluating pupil performance, the types of assessments that will be used to measure pupil progress towards achievement of the school’s pupil performance standards, the timeline for achievement of such standards, and the procedures for taking corrective action in the event that pupil performance at the charter school falls below such standards."

Such language is a useful start for developing an accountability plan that the school and its chartering authority, the school district, can agree on. However, in the first six years of this new law, the Colorado League has found that there has not been a common understanding from both districts and charter schools on how the schools are to be accountable, and in particular on how the charter renewal will be determined. Many have acknowledged that in too many cases the contracts between charter schools and their districts have been vague at best on these matters of accountability. (See CDE’s 1998 Colorado Charter Schools Evaluation Study: "Five schools said their districts had well-defined, written criteria for renewal. Another eleven schools stated that the criteria for renewal were not well-defined in their districts." Page 60)

The League’s Accountability Plan is designed to fill this gap, providing new charters and their districts with a process and criteria both can commit to—IN THE CONTRACT—that will clarify what is expected each year of the first five years of a school’s charter.

In addition, this Accountability Plan has stayed attentive to the new rules and regulations being developed by the state in implementing HB-1267, the accreditation of public school districts. The reporting done by and about these charter schools will include the kind of information HB-1267—and therefore all districts—will be eager to track and use as part of the evaluation of their schools.


C. Who will use this Accountability Plan?

This Plan is designed in particular for schools in their start-up phase and first five years.

1. For new schools being developed, it provides questions and criteria they can find useful in describing how they will be accountable to their chartering authority. The details will differ for each school, but the general process—building accountability around the Critical Questions leading up to external team visits in the third and fifth year—can provide charter developers a useful starting point for their application.

2. This Plan should also be of special interest to schools throughout their first three or four years, as we have detailed suggestions for the steps towards accountability that can be taken each year early in their existence, leading up to a site visit by an external team in their third or fourth year.

3. If this Accountability Plan seems to make sense for districts and charter schools during the first five years of the school, then it might have components that they could build on in establishing an agreement for the renewal application and the on-going life of the school during its next five years. Accountability is a never-ending focus, and recommendations here for years one through five might be borrowed and adapted for schools in their "second cycle," during their five-years of existence. For example, we believe that the whole notion of self-studies and self-assessment, followed by site visits conducted by external teams, is worthwhile for all schools. These two steps foster continuous improvement no matter how long a school has operated. For this reason we believe "older" charter schools that have been through renewal can continue to find merit in bringing in outside teams to offer an objective review of their progress. After its first renewal, charter schools could host such visits every five years or so, serving double-duty as visits to help the schools improve, while also providing districts with a useful process for on-going accountability.


Critical Questions

The Critical Questions the Colorado League has developed will guide much of this Accountability Plan. The five principal questions borrow heavily from those being used in Massachusetts and some other states that have developed accountability plans. These five questions are:

1. Is the school faithful to the terms of its contract?
   
2. Is the school faithful to its stated mission, goals, and objectives?
   
3. Is the academic program a success?
   
4. Is the school a viable organization?
   
5. Does the school recognize clearly where growth is most essential?

Critical Questions

1. Is the school faithful to the terms of its contract? Quite simply, is the school carrying out its contract with the district? Includes a look at the degree to which the school is following health and safety requirements, and the essential legal, statutory, and regulatory requirements to which it is committed.
   
   
2. Is the school faithful to its stated mission, goals, and objectives?
   
   
   1. What is the school’s mission? Are the school’s program and operations faithful to its stated mission, goals, and objectives, as stated in its charter?
   Related questions: Is the school design and educational approach clearly defined and coherent? Do the school’s organic documents—charter, educational vision, mission statement, guiding principles, etc.—reflect a coherent school design?
   
   
   2. What are the school’s goals listed in the charter application? Note any revisions made in the school’s goals since it opened. (Reserve examination of school’s academic goals for 3a-c.)
   If the school has non-academic goals for students (character, responsibility, behavior, leadership, citizenship, etc.) list those, too.
   
   Are these goals consistent with the school’s mission? Are they clear?
   
   Is the school meeting these goals?
   
   3. Is the academic program a success?
   
   
   1. Academic goals: Are the school’s measurable academic goals clear, and do they reflect high expectations?
      
      
   2. Assessments: How is student learning being assessed? Are the external and internal assessments in place well matched with the school’s academic goals? Are they aligned with the school’s goals to meet or exceed state or district standards? Are they aligned with the school’s mission?
      
      
   3. Results: Has the school made reasonable progress in meeting its own internally established measurable goals over its first few years of operation? Is student performance strong and/or are there clear indications students are making strong progress?
      
      
   4. What strategies are in place to see that students with special needs, those at-risk of failure, and those not making reasonable progress are meeting—or are being given opportunities and reasonable accommodations to meet—the academic goals? Are those strategies proving effective for these students in terms of improved academic performance?
      
      
   5. What kinds of curriculum and objectives and/or course outlines are in place? Are they clear? Are they consistent with the mission? Do they reflect high expectations?
      
      
   6. How does the school evaluate the effectiveness of its academic program?
      
      
   7. What changes have been made over the past year or two to improve the academic program? How are those changes being evaluated?
   4. Is the school a viable organization?
   
   Finances
   
   
   1. Is the school financially solvent and stable?
      
      
   2. Does the school have appropriate controls and procedures for the management of financial resources?
      
      
   3. Does the school commit its resources in ways that will help it achieve its mission and major purposes? Does it look like the school is making investments in staff and training, and in books, technology, and other supplies, that are consistent with the school’s priorities?
      
      
   4. Is a lack of financial resources impeding improved academic achievement?
      
      
   5. Is attendance strong? Is the school meeting attendance goals?
      
      
   6. Is enrollment stable and near capacity? Is the school showing good results on retention of students?
      
      1997-98 _____
      1998-99 _____        Waiting list currently is _____
      1999-2000 _____        Number of current students not returning _____
      
      
   7. Is school governance sound? How effective is the board?
      
      
   8. Is school day-to-day leadership strong? How effective is the school’s management/ administration? (Principal or school director; curriculum leaders; business managers; etc.)
      
      
   9. Is the school’s leadership—the combined work and joint efforts of the board and school management to lead the school—effective?
      
      
   10. What is the quality of teaching? Is professional staff competent and resourceful?
       
       Look into continuity in staff.
       
       
   11. What professional development is in place to support teachers in order to see that more students are meeting the standards?
       
       
   12. Is the school a place where students and teachers are safe and feel safe?
       
       
   13. Is the school climate good? Does it seem to be a positive learning and social environment? Is there a tone of decency and respect among and between students and the faculty?
       
       
   14. Is discipline handled effectively? Gather data for last two years on
       
       Office referrals_______ Suspensions _______ Expulsions ________
       
       
   15. Are the physical facilities adequate for the program of the school?
       
       
   16. Are parents satisfied with the performance of the school? What measurements are in place to indicate parent satisfaction? In what areas do they express concerns, and what steps are in place to address those concerns?
       
       
   17. How strong is parent involvement?
       
       
   18. How strong is the relationship with the chartering authority (district)? Is that going to be a factor in the school’s success and in its future?
       
       
   5. Does the school recognize clearly where growth is most essential? As the school analyzes its progress, how well can it identify its areas of strength, and areas that growth and improvement are most needed? Has the school established a limited number of vital priorities to focus on for self-improvement?
   
   

THE ACCOUNTABILITY CLAUSE AND SETTING THE STAGE
FOR A REALISTIC ACCOUNTABILITY PROCESS

Recommendation: Establish clear goals, and commit to a process for accountability in the contract with the district, but be careful about stating the exact measurements before the school opens. Most charter school applications and contracts do not contain clear language about the expectations of accountability and the means of evaluation and renewal. This creates a degree of unpredictability that is uncomfortable for both the charter school and the district.

A charter school's participation in this five-year Accountability Plan with clearly defined components, procedures, and documentation should facilitate predictable and effective communication between a charter school and the district. The charter school's commitment to the Accountability Plan should be drafted in contractual language so that all parties are clear about and legally bound to the stated responsibilities and expectations.

The process for adding an Accountability Clause to a charter school contract would begin during the negotiation process with the district or soon after a contract is signed (resulting in an amendment to the contract). The Accountability Plan described in this document offers a suggested process and timeline for accountability that is based on our work with charter schools over the past six years. If both parties agree to commit to this Accountability Plan, or a variation of it, the Accountability Clause would be written as an addition or amendment to the contract. It is suggested that this Accountability Clause or Amendment contain the following sections:

• INTENT
• EXPECTATIONS (Goals, Objectives and Pupil Performance Standards)
• DOCUMENTATION (School Improvement Plans, Annual Reports, Self-Study, External Report, etc.)
• TIMELINES
• CONTRACT RENEWAL
• COSTS

How specific should the goals and objectives for student performance be in the application? There are many unknowns as to how the charter idea in the application stage will transform itself into a real school community. In particular, the school will not know the capabilities of its student body until after the school opens and baseline data is gathered. For this reason, charter school applicants should set goals and describe plans in a general enough manner to allow for their lack of knowledge about their school community, yet specific enough to create understanding and trust with the chartering district. By avoiding too much specificity in the application, the school will have the needed flexibility as to the exact nature of the curriculum, assessments, and targets to shoot for in terms of improved academic progress. This will enable the school to make the inevitable adaptations in the first years of implementing the program. It is suggested that a plan to collect baseline data be presented in the application along with an agreement (described in the Accountability Clause) that the school will submit more specific and measurable goals for pupil performance by the end of the first year or beginning of the second year.

YEAR ONE/STEP ONE - A time for reflection about how the ideas expressed in the charter have met the test of reality in the first year of operations.

In the life cycle of a charter school, YEAR ONE is a time of frenetic activity. This time of creation does not allow much time for reflection. However, if the leadership of a charter school does not make time to reflect on its practices it runs the risk of creating quick fixes that might possibly compromise the charter.

Since the Critical Questions provide an effective overview of all aspects of the operations of a charter school, we encourage all first-year charter schools to review them. The Critical Questions for schools in their first and second years provide a framework to define and assess the operational areas and plans of the school. Sub-questions that were especially designed for first and second year schools are included to clarify the implications of the questions. We imagine a school would take these Critical Questions and spend much of Year One coming up with formal or informal responses to each question. Beginning to work on these questions early will assist the school with the completion of the Self Study by the end of Year Two, as well as help them prepare for their Third Year Site Visit.

At some first year schools an internal examination of the Critical Questions might serve as a sufficient means for them to reflect on their practices and operations. Some schools might want to invite an external person to facilitate their internal review. In addition to an internal review of the Critical Questions, some schools might need technical assistance or facilitation with some specific aspects of their operations during the first year. New schools often lack the personnel to effectively define, build, manage, and assess all of the various components (financial, legal, human relations, educational, governance, etc.) of their operations during the first year. The Accountability Plan outlines possible options for technical assistance or facilitation that are based on the League’s work with first year charter schools and what it has learned about their developmental needs.

The League’s experience with charter schools has shown us that many first year schools would benefit from technical assistance in the form of: 1) Goal Setting Facilitation, 2) Governance Review, and 3) Administrative Review. Some schools may decide to seek technical assistance in one, two, or all three areas.


YEAR TWO/STEP TWO – internally applied data gathering and self-study in preparation for a year three site-visit/formative evaluation.

Good schools are eager to ask themselves hard questions: are they fulfilling their mission? Are they becoming the kind of school they said they would be? Are they serving students and parents as they intended? Are their students learning and growing in a way that meets high expectations? And much more.

The League’s Accountability Plan creates a useful framework for this kind of analysis. It also ties this effort to wrestle internally with these hard questions to a site visit the following year from an external team. The product of the school’s self-study will be a notebook that responds to the Critical Questions. We recommend that a nearly final draft of that self-assessment be completed by the end of the school’s second year. A final draft—ready to go out to an external team—would be completed and ready by the beginning of the school’s third year.

Some worry about the duplication of an effort like this when schools are already doing considerable data gathering of their own. But again, our proposal anticipates that this is simply the format for gathering and communicating information that the school and the district have already agreed to in the initial contract. None of this will come as a surprise; it will be understood from day one that an internal review, built around the Critical Questions, would take place during the second year.

What is more, we believe the Critical Questions we have designed address most of the key issues districts and schools will want to examine to evaluate progress. If there is something not covered nor deemed necessary by these questions, then again, in the initial contract, both parties can negotiate other items they want to add, or delete, other information they want, or do not want, reported. The point is that the contract should prevent later misunderstanding or arbitrary changes in what criteria will be used to measure the school’s growth and effectiveness.

While we know that a good self-study does take time, we want to emphasize that this effort should build on all the information the school has already gathered or is already producing as part of its regular reporting to parents and to the district. These reports naturally include the Annual Report to the Community at the end of the first year and the School Improvement Plan for the second year. Of special interest will be the concrete data that might be part of those documents—from assessments on student performance, parent surveys, and other efforts during the first year to evaluate effectiveness and satisfaction. By pulling much of this information together around the Critical Questions, we think it will mean all the data-gathering is well–directed, and will lead to a richer, more comprehensive self-assessment than most schools in their second year typically produce.

Finally, this self-study is designed to help a school really know itself better, to help a school see if it is indeed becoming the school it set out to be. And the Critical Questions will remain a constant guide. We have phrased some questions somewhat differently to help schools for this self-study (see Critical Questions for Self-Study), but the topic of each question is the same. When—a year later—a visiting team comes and provides a formative evaluation of the school and its progress, the school can be confident that this outside evaluation, just like its own internal review, will focus on these same questions.


YEAR THREE/STEP THREE – low-stakes site-visit by an external team, with school improvement process very much the theme; a kind of formative evaluation, offering both commendations and recommendations for improvement.

During the third year a visiting team of five to seven educators will spend two days or so at the school "validating" the school’s own-self-study, and putting together a formative evaluation*—again, all built around the same Critical Questions the school addressed in its self-study. The team will seek to find out to what extent the school is accurate in its self-portrait, and how well it has defined its strengths and areas of concern.

The final report will conclude observations, commendations, and recommendations. The recommendations will be offered as ideas and approaches the school should consider, but not as dictates that must be implemented. Visiting teams are aware that in only two days they cannot truly know a school, and they will present their recommendations with the understanding that this is advice they hope will be given some real thought—but not with the presumption that they have the right answers.

____________

*A formative evaluation provides information and insights to help a school improve.

A summative evaluation—see Year Five/Step Five—determines if reasonable progress is being made based on an evaluation of the school’s goals, its formative evaluation, and its current state of operations.
____________


The role of visiting team members is to observe, to listen, and to validate what the school has said in its own internal self-assessment. The purpose of these visits is largely to help a school see how well it is doing in meeting its own mission. The role of team members is NOT to come in and encourage the school to become more like their respective school they are from. With proper training and/or reminders from the team leader, all team members will demonstrate respect for each school’s mission and goals. Team members realize that they are expected to work hard to see the school for what it hopes to be.



To be effective, these site visits depend a great deal on effective leadership from highly capable team leaders. The League has drawn from an experienced group of well-respected educators to lead the ten or so site visits it has overseen during the past three years—including six site visits in the spring of 1999. Most of those chosen are currently school directors themselves. Many have taken part in external studies in the past as part of accreditation efforts either in schools abroad or in the United States. The League provided training to team leaders prior to its spring 1999 visits. This current plan outlines even more careful training for any new leaders recruited for such an important task.

Third year visit — formative evaluation — NOT to be used for charter renewal

We believe a report from an external team that visits a school during the first half of its third year should not be used as the basis to determine renewal. It is too early. Most new organizations take much more time than this to solidify their program and operations. While this Accountability Plan seeks to help districts and charter schools come up with a process that strengthens accountability, we believe that a third-year visit should play a key part in that process—but that it should not be used as part of the summative evaluation that will guide a district’s decision whether to renew or not to renew. If used for that purpose so early in a school’s history, it changes the nature of these site visits in a fundamental way. In order for a school to grow and learn from its struggles it must feel free to openly discuss them and seek the opinion of those who are outsiders and who can offer objectivity. However, it is difficult to be honest and open about your difficulties if you are fearful that the information you share could potentially be held against you in a decision about the renewal of your contract.

Instead, as part of a five-year accountability process, we see the report from the third-year visit as designed especially for the school and its improvement. Some worry that if these reports also go to the school district it might upset the dynamics one wants in such visits, a healthy spirit of openness and honesty—with a focus on learning from an external team what appears to be going well, and where improvements might be considered. Another view, however, is that the chartering authority for each school should also hear of the school’s progress to date, and see a copy of the team’s report. There is probably no right or wrong approach. The League proposes that the district and school discuss these issues and resolve them up front. If the district gets to see a copy, and its findings are fairly negative, we trust districts will understand that these visits and reports are intended to assist the school in making sufficient progress to earn renewal by year five. With that understanding we would hope that critical findings in this first external evaluation would not jeopardize a school’s existence.


YEAR FOUR/ STEP FOUR – the school reflects upon, writes a response to, and takes action around the third year external study. Prepares its self-study in readiness for a Year Five, high stakes external study that will be part of the renewal process.

Year Four is the time the school has the opportunity—we would add, the responsibility— to make use of the insights and recommendations from the visiting team’s report. It would be disturbing, and perhaps disastrous, if the report of the Year Three external team were not adequately analyzed and acted upon. This Accountability Plan puts some mechanisms in place to be sure that the school addresses those elements of the program, the leadership, the financial well-being—or whatever it might have been—that the first visiting team saw as weaknesses or matters of concern. Such mechanisms do not guarantee improvement, but they will at least make sure the recommendations from the Year Three site visit are not ignored. We hope, of course, for much more than that; by taking these steps the school should make real progress on items the Year Three visiting team believed to be in need of improvement.


A YEAR FOUR SELF-STUDY

During Year Four the school will start to prepare its report to the district and charter renewal visiting team for Year Five, a report probably due in the fall of Year Five prior to any team coming to evaluate its progress. This report—in most cases another self-study of sorts, depending on what the district asks of it in the initial contract—will assess where the school is as it completes its fourth year. The school should make references back to what the school said about itself in its own Year Two self-study, and what the external team found in its Year Three report. It would be useful to note progress in areas where concerns were expressed in that Year Two self-study and Year Three external report.

In the initial contract, districts and charter schools would want to consider instituting a couple of means of additional follow-through from the year three visit. Everyone is familiar with the worry about external studies collecting dust—and not having a tangible role in a school’s continual improvement efforts.

1. The district might very well want the charter school to report on the steps it has taken around the recommendations from the external team. To this end, the contract could require that a school write a short reaction report (three to five pages) during Year Four on how it has used the Year Three formative evaluation, what recommendations it has acted on and how, and what recommendations it did not choose to implement and why. This report could be sent—again, depending on the initial agreement signed by the school and district—to the team leader of that site visit from the previous year, as well as to the district.
   
   
2. The contract might see a role for the team leader from that site visit, a district representative, and/or the organization that coordinated the site visit (such as the Colorado League), so that someone follows-up with a meeting at the school to review its progress since the external team came in. In this way an outside person would help ensure that the findings of the site visit are still "on the table," and that the recommendations from the external team have not been put "on the shelf."

Schools will benefit from having a clear understanding of the information the district and the charter renewal visiting team will be seeking Year Five because, once again, the same Critical Questions will guide that visit.


YEAR FIVE/STEP FIVE – high-stakes site visit/outside evaluation/validation as part of the charter renewal process, using in large part the progress demonstrated around the Year Three evaluation and recommendations.

As the chartering authority, school districts are responsible for evaluating their charter schools. We recommend that districts use something very similar to the year three visit. While each district may be asking for some additional information and may wish the teams to look into a couple of items not included in our Critical Questions, the heart of the visit and the external report should again be on these same Critical Questions. Once more, this is to create consistency, and to assure the schools that the criteria for renewal—in line with all they have done towards accountability and improvement during their first four years—will not be a surprise.

A Year Five evaluation can be built around the same Critical Questions used by the external team Year Three. (The League has not developed a format designed for Year Five.) All that is described in detail in our Year Three/Step Three notebook for Site Visits and External Studies provides the basic framework for the Year Five site visit.

There would be a difference in intent, however, as in this case the primary purpose would not be a formative evaluation since baseline data was collected during the school’s first external site visit. The Year Five site visit would produce a summative evaluation, and would help determine if reasonable progress is being made based on an evaluation of the school’s goals and internal report, its Year Three/Step Three External Report (formative evaluation), and its current state of operations. Based on the findings and recommendations of that earlier visit, we expect some questions might have greater significance than others for the district and the school. Such a fifth-year visit might zero in especially on those particular questions. The goal is to follow-up on all that was learned about the school Year Three, so as to help weigh the progress of the school over these past two years leading up to renewal.



We think that for a summative evaluation as part of the renewal process it is especially important that districts follow our model for Year Three and use an external team for the site visit. Jefferson County moved to the concept of external teams in the fall of 1997 as part of its charter renewal process. After three such site visit teams during the 1997-98 and 1998-99 school years, the district decided to endorse this approach as part of its process for renewal. A team avoids the limitations of having one or two people evaluate the charter school, and an external team avoids the potential for conflicts or political concerns when district people evaluate one of their own schools.

Outline of Accountability Plan (for schools with five-year charters)