Propaganda Can't Save Schools: How will we know if Common Core has "worked" in Kentucky?

2014 was the year we were all supposed to know whether the Kentucky Education Act of 1990 (KERA) had worked. And yet here we are, in 2014, talking about a completely different education reform proposal: Common Core.

KERA is dead: Long live Common Core

All experiments have some procedure for evaluating their success--some means of verifying whether a particular program is successful or not. The advocates of Common Core owe it to the rest of us to explain exactly when and how we will know if their program has worked.

With KERA, which involved the largest tax increase in state history, the justification of which was to improve our schools, we were given an actual date and some actual numbers that would verify its success. We were told that by 2014, all schools would reach "proficiency" on state tests. We were even given exact numbers: "Proficiency" meant scoring 100 to 140 on state tests. If this happened, then it would be judged a success. If it didn't happen, then it would be judged a failure.

State Sen. Ed Ford, KERA advocate and then chairman of the Senate Education Committee famously said, "It will take a generation to know whether KERA has worked." Well, it is now 2014 and not only did KERA not pass its own self-imposed test, but it has been almost completely forgotten.

By its own criteria, KERA was a failure.

But at least there were was an actual date and actual criteria. Where is the date we can use to tell if Common Core has worked? Where are the criteria we can use to judge its success?

Brad Hughes, spokesman for the Kentucky School Boards Association, says its "too early to tell" if Common Core has worked. He says two years is too short a time period to tell whether it has succeeded. We need, he says, "at least three years worth of data to tell."

Okay. Three years to tell if Common Core has worked. Is that our time frame? Is everyone agreed on this?

But then we still have the other matters necessary to verify success, such as 1) having criteria we can use to determine whether a school has successfully implemented Common Core; 2) a means of telling whether a school's improvement is due to Common Core or some other factor; and, most importantly, 3) Specific criteria whereby we can define what success is.

Where are these specific criteria? Can Hughes tell us? If not Hughes, who?

We need to get beyond the propaganda. The failure of KERA should have taught us that propaganda can't save schools.

HT: Richard Day

Prichard Committee challenges the CJ on the facts about Louisville schools

The Prichard Committee is calling the Louisville Courier-Journal on the carpet for factual inaccuracies in its editorial last week attacking the Jefferson County School Board for firing Superintendent Sheldon Berman. Prichard apparently thinks the CJ has some obligation to truth in its editorials.

We're not sure where Prichard got that idea.

The CJ, which on education issues plays Charlie McCarthy to the Jefferson County Teachers Association's Edgar Bergen, had stated that "percentages of students testing proficient in basic academic skills have risen steadily." Not so, says Susan Weston at Prichard:

I respectfully submit that since 2007, proficiency levels in Jefferson County Public Schools have risen steadily only in high school writing. In every other tested subject at every level, proficiency declined in one or more of the last three years.

And that's not all:

I turn to another claim in the same editorial, this time the one that says "students rated novice have dropped sharply."

For that statement as well, I respectfully submit that the editorial has not accurately described the facts.

Far from dropping sharply, the percent of Jefferson students scoring at the novice level increased from 2007 to 2010 at every level in reading, mathematics, science, and social studies and at the elementary level in writing. The only novice results that have a net three-year decline are middle and high school writing, and while the 14.5 percent decline for high schools is a large one, the 0.24 percent shift for middle schools is not a drop to which the modifier "sharp" can reasonably be applied.

Ouch.

This is interesting (and a little ironic), since, in the old days, it was the Prichard Committee who played the role of Tokyo Rose in the war on ignorance in Kentucky. When I was the one battling Prichard on education issues, they were the ones misleading the public on education issues, trying to conceal problems in the implementation of the Kentucky Education Reform Act (KERA) and actively trying to suppress the public release of important information showing problems with the state tests.

Could it be that they have now come over fully from the Dark Side?

The Prichard Committee in Wonderland: When logic and proportion have fallen sloppy dead

In response to my recent post "The Death of KERA," Susan Weston at the Prichard Committee Blog characterizes me as "spinning wondrous tales" over the years about a KERA dragon that ravages the land eating educators caught teaching spelling and setting fire to villages caught administering multiple choice tests. The real story of KERA, she says, "has been dull by comparison."

I've heard KERA called a lot of things, but this may be the first time it has been called "dull." After disagreeing with the dragon narrative, Weston relates her own narrative--one in which everyone lives happily ever after:

KERA delivered stronger and fairer school funding, reduced political corruption, and vastly improved facilities and technology. It nurtured more focused teachers, better instructional leaders, and a big step up in justified pride in public education. We've still got work ahead to strengthen classroom work, not because the primary program, extended school services, or sustained professional development were mistakes, but because we didn’t put in the hard work to help them succeed.

I'll leave it to readers to determine which is the fairy tale.

I don't remember actually comparing KERA to a dragon, but I do remember comparing it to Alice in Wonderland. In fact, my career as a prominent KERA critic back in the 1990s began with a public debate in the Danville newspaper with none other than Susan Weston, a debate in which I compared the rhetoric about all students being equal to the caucus race in Lewis Carroll's book, an event in which everyone wins--and everyone get's a prize.

That was only one of the silly and sometimes surreal practices that were foisted on schools when the reforms were implemented. It was a bit like being in a Jefferson Airplane song. There was best guess spelling, and the new New Math, and open classrooms, practices most of us thought were discredited in the 1960s, but which those implementing KERA thought the rest of us had forgotten. The only thing missing was tie-dye T-shirts and peace signs.

I remember a retired superintendent calling me one day after something I had said in the newspaper. "You're absolutely right," he said. "We had just put the walls back up in our building that they had torn down when we were doing open classrooms, and then we had to tear them back down again for the non-graded primary program."

Oh, and I didn't see anything in Weston's tale about the year that, under the KIRIS testing system (the precurser to CATS), the best school district in the state (Anchorage Independent) was rated the worst. Not exactly a result that comports with logic and proportion.

It's a story I've told many times, but just for old time's sake, I'll tell it one more time. I went to an inservice day in 1992 at Lawrence Dunbar High School with a friend of mine. The Department of Education presenter approached the podium and began her harangue about the evils of traditional education techniques and explained why we needed to replace them with the "new" practices under KERA.

"When you learn," she asked, "do you sit in straight rows of desks, sitting under phosphorescent lights listening to a lecture? Or do you do it better sitting back on a couch, with the sun coming through the window, and talk with your best friend?"

Exactly how we were going to provide this experience for the tens of thousands of school children in Kentucky wasn't exactly clear, but I remember leaning over to my friend and whispering, "Look around the room." We were surrounded by teachers and administrators sitting in straight rows of desks, under phosphorescent lights, listening to a lecture.

To make any sense of that, you'll have to go ask Alice (when she's ten feet tall).

Or maybe I'll just take Susan up on her offer to visit her at her favorite hangout: Danville's "Hub." I suppose it is fitting that the debate over KERA should begin and end with the same two people jousting over education policy. I might even try those "magic free muffins" she mentions they sell there--although I'm not entirely sure, given her fantasic account of KERA, that those muffins don't have something baked into them.

Does the death of KERA mean the death of education reform?

Richard Day has a very informative post on his blog Kentucky School News and Commentary about the reaction to the now deceased CATS testing system. But in the otherwise fairly accurate post, he interprets me as disagreeing with Dan Kelly's assertion that the end of the CATS tests was not the end of education reform in Kentucky. But I did not say that education reform in Kentucky was dead: I said that KERA was dead. Richard fails to make that distinction.

If education reform and KERA are synonymous, then we're in even bigger trouble than a lot of us thought.

The Death of KERA

The Kentucky Education Reform Act of 1990 is now officially dead. The program was made up of many components, some of which have effectively been abandoned (like the nongraded primary), and others that live on (the family resource and youth services centers), but the heart of the program was always the testing system.

And when the heart goes, so does the body.

You could call this a transplant, of course--taking out a high stakes test that measures the performance of schools, not students, that uses unreliable open response questions rather than multiple choice questions, and that grades schools on subjective portfolio assessments, and replacing it with one that doesn't do those things. But it hardly seems worth the trouble to attempt it.

No. This was turning off the life support machines.

The irony of this whole thing is that those of us who fought this back in the 1990's advised policymakers to do exactly what they did today: to drop portfolios from the accountability for schools, to drop the ridiculous open response items that have not (Repeat: have not) improved writing, and give parents a test they can adequately judge the progress of their children with.

We were told we were against education. That we were not hip with the educational times. That we were not familiar with the educational research (that was really just trendy pronouncements) that said this stuff would work. That we were opposing progress in schools.

Funny. Do you hear any of that now? What was the House vote? 93-0?

I suppose I should feel vindicated, but I wonder about the Lost Generation: the children who went through the KERA system who were denied a proper grounding in basic skills in the nongraded primary program. Who were told they were learning to write but who were instead denied help in grammar and spelling by teachers who, because they were told they couldn't, were scared to say anything. Who thought they were learning to read, but denied help in sounding out words because it was discouraged by whole language advocates.

What about them?

In his book The Structure of Scientific Revolutions, Thomas Kuhn argues that movements don't die because they were repudiated; they die because the leaders of those movements themselves die. We may have the same thing here. How many of the people who voted for the Kentucky Education Reform Act of 1990 are still in the legislature? I'm thinking it is less than 20.

One of the people in the legislature when KERA passed was Ed Ford. Ford chaired the Senate Education Committee, helped shepherd the bill through the legislature, and assisted in its implementation. Ford one proclaimed that "it would be a generation" before we knew whether KERA had worked.

Well, a generation has passed. And we know now, don't we? Who ever thought that such a momentous action as was taken today would have been attended with such little fanfare? I'm told that that's not uncommon when they turn off the life support machines of a dying patient.

Will the last person out of Kentucky's Education Reform Headquarters please turn off the lights?

Education bureaucrats caught taking credit for private school success

Another Kentucky Department Education KERA success claim exposed. For years the Kentucky Department of Education has reported ACT scores that didn't look half bad. But the data they reported didn't separate out private school scores. Some of us have wondered all along whether the scores that the Department was reporting didn't make make public schools look better than they were by incorporating higher nonpublic school scores.

Well, whadya' know.

Dick Innes at the Bluegrass Institute got his hands on the numbers which the Department apparently didn't want to publicly release. Here they are:

Private schools outperform public schools in all years but one, which Innes attributes to an influx into the private schools of less well-prepared public school students.

Read the rest of his analysis here.

The death of the CATS test

Like other felines, CATS seems to have had nine lives. Since 1992, when the KIRIS tests made their debut, the testing system has proved the most controversial aspect of Kentucky's education reform. Time after time the test has taken hits for its inaccuracy, unreliability, unmanagability--not to mention its sheer intrusiveness in the education process.

If you want to know what it has been like for those of us who have tried to stop the nonsense all these years, just watch the scene in Star Wars where they try to attack the Death Star: the thing is just so big and seemingly invulnerable that every shot just bounces off.

I'd love to say that those of us in the Rebellion delivered the final blow to the thing by finding a vent somewhere where we used the Force and got it down the right hole, but in reality what has happened with the test is that it just plain petered out.

Was it Thomas Kuhn who said that intellectual revolutions come about not because one theory is refuted by another, but because the advocates of the reigning theory simply die off?

That may, in fact, be the situation with the CATS tests: those who swore the blood oath in 1990 to defend every aspect of the reform act to the death just faded away. How many legislators who actually voted for KERA are left? And isn't it an irony that one of the few left is the one who is bringing the test down?

This is the way the test ends:
This is the way the test ends:
This is the way the test ends:
Not with a bang, but with a whimper.

The Prichard Committee backfire

The best thing to do to head off policy changes you don't like is to call for a "study" of the issue. When Paul Patton was almost defeated by Larry Forgy in 1994, he created a governor's task force to "study" KERA. I know. I was on it. Forgy had called for changes in the reform law, and the KEA controlled Patton administration knew they had to at least look like they were doing something.

Now, with several bills looming in the legislature, Gov. Steve Beshear is, once again, calling for a study. What will a study tell us about that 18 years of actual experience in implementation hasn't?

If we don't know what the problems with KERA are a generation after it was begun, then how exactly is a study going to help us?

Oh, and does anyone remember State Sen. Ed Ford's statement that it would take a generation before we could know if KERA has worked? Well, it's here, folks.

No one knows how much CATS costs, committee finds

Oh dear. While some of us were thinking that the problem with CATS was that it costs too much, all of a sudden we find out that that's not the problem. The problem is we don't know how much it costs. Turns out that our wonderful education bureaucrats have no idea how much the monstrous state education testing system is setting us back because of poor accounting.

All state auditors could determine is that it costs at least $18.6 million, a higher figure than has been reported before. But there is no way, given the state's poor accounting, to know the total cost of the tests because no figures are available to determine how much local school districts are spending, and amount that is likely to be very high.

"There isn't a mechanism to be able to determine the cost at the local level for the assessment testing," Brian Lykins, director of special audits in the auditor's office, told the Louisville Courier-Journal.

This comes at a bad time for supporters of the tests, since President of the Senate David Willliams has announced that he would like to see the test eliminated.

Stay tuned on this issue...

David Williams takes on CATS

I still haven't seen it yet, but on KET's "Kentucky Tonight," State Senate President David Williams apparently called for dumping CATS and changing math education. Williams voted for the Kentucky Education Reform Act of 1990 (KERA), so he can't exactly be labeled a fire-breathing anti-KERAite. Good credentials for the person who takes on some of the excesses that still plague Kentucky schools as a result of what many of us at the time labeled a boondoggle.

KERA simply never has been able to live up to the hype which has always surrounded it. Has there been progress in Kentucky schools? Here and there. But, as Larry Forgy pointed out at the time (and he was one of the plaintiffs in the case that prompted the State Supreme Court's Rose decision), there were a lot of good things already afoot before KERA turned the state's schools upside down.

The question is whether Kentucky schools would have been better off without KERA. We'll never know the answer for sure, but it wouldn't be a hard case to make that the entire generation of children who were robbed of a decent education in basic skills by the now somewhat etiolated nongraded primary program were ill-served.

And then there was the preposterous attempt to teach math using "math essays," thanks to "math portfolios" in which students were not even allowed to write numerals, but were to spell out each number. And the ludicrous attempt to teach writing by preventing teachers from telling kids how to correct their mistakes, thanks to the writing portfolios. We could go on.

The chief role of CATS has been to hide from ourselves the consequences of this interminable educational silliness.

Williams faces an uphill fight since there are still people who have a political stake in propping up KERA. But surely there will come a time when the Fathers of the Kentucky Education Reform Act have died out, and more level-headed generation takes its place.

May that time come soon.

Discussion on state education testing now online

My televised discussion/debate on education testing with several other figures in state education in Kentucky that took place last Monday, June 23 is now online at Kentucky Educational Television (KET). You can access it directly by clicking here. Guests on the show were:

• Sharron Oxendine, president of the Kentucky Education Association
• Lu Young, superintendent of Jessamine County Schools
• Tim Decker, an art teacher at Russell Middle School
• Martin Cothran, senior policy analyst with The Family Foundation of Kentucky

Is there hope for the state testing task force?

For Immediate Release
June 11, 2008

Contact: Martin Cothran
Phone: 859-329-1919

“Don’t rock the boat” the theme of state testing
task force says family group

LEXINGTON, KY—“If the state education testing henhouse needed foxes to guard it, they got the best ones available,” said Martin Cothran, senior policy analyst with The Family Foundation of Kentucky about today’s announcement of the membership of a state task force to review the state’s CATS testing system. “We’re just going to call it the ‘Status Quo Panel’,” said Cothran.

“’Don’t rock the boat’ seems to be the developing theme for the Status Quo Panel,” said Cothran. He argued that there is only one panel member who has been publicly critical of the testing system: State Sen. Dan Kelly (R-Springfield). “In fact,” said Cothran, "the legislative members of the panel are almost two to one Democratic.”

Cothran said his group didn’t know every member of the new panel, but said if the ones who have been involved in past education debates were any indication of the rest of the panel, the signs were not good. He pointed to the appointments of Bob Sexton of the Prichard Committee, former Patton administration State School Board Chairman Helen Mountjoy, and Sharron Oxendine of the Kentucky Education Association as examples of panel members who could be counted on to argue against substantial change in the CATS testing system. There is also only one member who represented parents, said Cothran, and no testing experts.

“The Status Quo Panel appears to be sailing with a cargo full of apologists for the testing system.”

“The task force has some great leadership on the side of not rocking the boat,” said Cothran. “The point of this task force was to take a critical look at the testing system. But in order to take a critical look at something, you have to have critics. Other than Sen. Kelly and Sen. Vernie McGaha (R-Somerset), who on this panel is actually going to be critical of the system?”

###

Charles Murray on a modern education heresy

I've always said that a couple years in teacher's college is as good as a lobotomy, and the woolly-headed thinking about human nature that gets propagated there is the major cause of our education woes.

One of the unquestioned dogmas that gets passed on to each successive generation of teachers is that "every child can learn at high levels". This was one of the mantras of the Kentucky Education Reform Act of 1990, one of the most sweeping education measures ever approved by a state. And in the early 1990s, it was considered heretical to question it. All children are equal, and all have equal natural capabilities.

Charles Murray is the perfect person to address this issue, as he does in this article, with his background in dealing with the issue of nature and nurture, a debate the fire of which he threw a great deal of gasoline on in the late 1990s in his book The Bell Curve. Murray just reported the data, and the liberal media proceeded to do two things: either scream bloody murder, or stick their collective heads in teh sand.

What made the news was the very brief section in the book about intelligence and race, a rather unremarkable part of the book anyway. The real point of the book was to report what the evidence tells us about how both nature and nurture affect who we are and what we become. Murray and Herndon, the co-author, delivered the common sense conclusion that nature determined from 40-60 percent of what we become intellectually, and nurture the rest.

In this essay, Murray asks why it is that the education establishment just simply ignores this fact. I'll have more to say about it in the weeks to come.

The problem with education is not money

Nicholas Stevens at Cyberhillbilly has this to say about the demise of the casino bill in the Kentucky legislature:

The casino bill is dead, what is not dead is the realization of education in the state with being overall underfunded and not performing to national standards, which even our national education standards are behind.

Let's see now. We were whining about education being underfunded in the 1980's in this state, so we instituted the lottery in 1988. So what were we whining about in 1990? Not enough education funding. So we passed KERA, the financial part of which was the greatest increase in state history. So here we are again. And the problem? Not enough education funding.

Exactly what do we have to do to please the people who say we are not doing enough to fund education? We currently spend about 65 percent of the state budget on education. How much of the state budget is it going to take to satisfy them?

The answer to the question, of course is: No amount of education funding will cause the people who say there is not enough education funding to stop calling for more.

The irony is that the problem with education is not money. For one thing, there is little correlation between more money spent and better education. Not only are the states with the best education systems necessarily the ones who fund it at higher levels--nor are the ones with the worst systems the ones who spend the least, but private schools, who spend less that public schools per student, generally outperform public schools.

Could it be that THE PROBLEM WITH EDUCATION IS NOT MONEY? Maybe the biggest tax increase in Kentucky history did not produce what we hoped because of what was done with the money. Instead of doing things that have demonstrably worked in education, like a greater stress on basic skills and a greater emphasis on solid content (and less on amateur psychology), we spent the money on warmed over experiments from the 60s, like nongraded primary programs, whole language, and the newest version of the New Math.

I say we put a moratorium on new money for education until we have some assurance that the money now being given to schools isn't being misspent on the newest educational fads and gimmicks.

KDE fudges numbers in new audit report on writing portfolio

Well, the Kentucky Department of Education strikes again. Just as we publicly released the 2005-2006 Writing Portfolio Audit Report (because they didn't), they release the 2006-2007 Audit, which makes it look like there has been improvement in the scoring of portfolios, which are a significant part of the way schools in Kentucky are assessed and held accountable.

We had gotten a hold of the 2005-2006 audit of the portfolios which showed that a startlingly high percent of them were graded higher than they should have been. Why is this significant? Because it calls into question the argument that Kentucky students are doing as well as proponents of the 1990 education reforms say they are.

But on his blog Kentucky School News and Comment, Richard Day responds to my post. His first argument against my statements was that my characterization of the audit as "closely held" was inaccurate:

Cothran's suggestion that the audit reports are "closely held" seems to lack justification. KSN&C asked for and received the current 2006-2007 report from KDE without objection or inquiry within an hour this morning. By law, KDE must audit portfolios annually to monitor the degree of inter-rater reliability and the report is public record.

Okay, several things: First, I didn't say that KDE was unwilling to release the report when asked, or that anyone had to pry it out of their grip by force. Secondly, I wasn't talking about the 2006-2007 report, but the 2005-2006 report. So I am still a bit mystified as to what was wrong with my characterization of the report. I chose the expression very carefully: I said "closely held" because very few people outside of KDE were aware of its existence until we outed it. No legislator was made aware of it and it was absent from their website.

The question isn't whether the audit was closely held: the question is why it was closely held. Here we have been arguing over the fate of the CATS test for weeks and KDE has a report which bears directly on the issue and we're the ones to end up having to make it public. What gives?

But then Day gets directly to the issue of reliability, admitting it is a problem, during the course of which he quotes KDE apologist Lisa Gross:

KDE spokeswoman Lisa Gross told KSN&C,

As students become more refined writers, it is sometimes more difficult for two scorers to agree completely on their ability levels. The complexity of their work can cause a score to be "on the edge," meaning that another scorer could consider the work to be of slightly higher or slightly lower quality...What the overall audit results tell us is that we need to enhance scorer training and ensure that everyone is comfortable with the analytic model.

The results, and human nature, might also imply that close calls get bumped up. The only way to even approach reliability is through carefully described process.

And how long has KDE had to do this? Sixteen years. So what was the hold up? And why is it so easy to "bump up" the portfolio scores? Could it possibly be because THEY ARE COMPLETELY SUBJECTIVE!!!

Then Day argues that the process of grading has been changed, implying that this has solved the problem with the scoring. But has it been solved? One would think, if it was, then the results would be appreciably different in later audits. And, indeed, if you were to look at the figures Day posts on his blog, you would get that impression. Day posts the pages from the 2006-2007 audit which was released the same week as we released the 2005-2006 audit.

Lo and behold! The agreement between the original portfolio scores and the audited scores looks much higher! In the new 2006-2007 charts there are now figures of 90%+ where in the 2005-2006 there were figures as low as 8.89% They must be doing a better job of scoring! You would think so, anyway, given the way it is presented.

Think again.

Day uncritically posts the pages of the new audit without pointing out that, in an apparent attempt to hide the fact that the new audit shows that the problem is just as bad as it was the year before, the new audit changed the way it reported the figures. Whereas the 2005-2006 audit reported the percentage of portfolios that received the same ranking (Novice, Apprentice, Proficient, Distinguished) by the original grader and the auditor, the new audit reports portfolios that received the same grade or the next grade above or below it.

In other words, if one of the 2005-2006 portfolios received a "distinguished" ranking by the original grader but only a "proficient" grade by the auditor (one full step down), it was counted as having been scored differently. But in the 2006-2007 audit, the same situation was not counted as having been scored differently: it was considered to have been close enough. In the original audit, a portfolio was considered incorrectly graded if it was scored differently. In the new audit, it has to have been so far off as to be practically an act of incompetence to count as being scored incorrectly.

Look at the headings on the charts. On the 2005-2006 Audit, it says "Percent Agreement." But on the 2006-2007 charts it says "Percentage Agreement Exact and Adjacent." Heck, why don't they just say, "Percentage Agreement Exact and Inexact"? It would be no less accurate a characterization.

Now this is a fairly transparent attempt by education bureaucrats to make it look like there has been improvement where, in fact, there has been none. If you look at the actual numbers, they are about as bad as they were the year before. Fortunately, Mark Hebert of WHAS-TV in Louisville reported the numbers correctly, observing that they were just as bad as the year before.

What is ironic about this is that what KDE is doing is trying to hide the fact that portfolio scores (and therefore CATS scores themselves) have been inflated. And how do they accomplish this little exercise in deceit? By inflating scores!

I have dealt with KDE cant and deception for 18 years now, so it doesn't surprise me a bit. I lost any respect I had for these people a long time ago. But what does surprise me is that there are still people out there who fall for it.

Now Richard Day is a fair and honest guy, and I'm assuming he just didn't notice the fudged numbers, or didn't see it as necessary to not the difference between the way the two audits reported the numbers. But I can't resist throwing back the words he threw at me in his post, and point out that any belief that the scoring has improved "lacks justification."

Now Richard knows why he got that new report with so little coaxing.

CATS portfolio scores overstated, says audit

For Immediate Release
Contact: Martin Cothran
March 17, 2008
Phone: 859-329-1919

LEXINGTON, KY—A closely held portfolio audit conducted by state education bureaucrats calls an important aspect of the state CATS tests into question, according to a family advocacy group supporting proposed changes in state education testing. “This report is a blow to the idea that the CATS tests are a reliable indicator of how our students are performing,” said Martin Cothran, senior policy analyst for The Family Foundation of Kentucky. “The portfolios are a key element of the state CATS tests. If this audit says what it looks like it says, then we’ve got serious problems that lawmakers cannot afford to ignore.”

The Family Foundation is supporting Senate Bill 1, which would replace the CATS testing system with a more objective, easier to administer and grade multiple choice test that would give reliable scores for individual students.

The audit, conducted by the Kentucky Department of Education (KDE), shows that writing portfolio grades given by CATS graders in 2005-2006 were dramatically higher than they should have been and that the higher the grade given, the less likely it was to be an accurate grade. Some agreement rates between how the portfolios were actually graded and how they should have been graded were lower than 20 percent.

KDE Portfolio Audit Agreement Rates¹

	Novice	Apprentice	Proficient	Distinguished
Grade 4	100% (24/24)	89.42% (186/208)	64.97% (230/354)	28.74% (25/87)
Grade 7	98.89% (178/180)	83.45% (237/284)	42.01% (92/219)	8.89% (4/45)
Grade 12	96.88% (31/32)	70.11% (190/271)	38.92% (65/167)	18.92% (7/37)
Total	98.73% (233/236)	80.34% (613/763)	52.3% (387/740)	21.3% (36/169)

“The audit seems to suggest,” said Cothran, “that about 75 percent of the portfolios ranked as “distinguished” in 2005-2006 were graded too high, and almost half of portfolios rated “proficient” were given higher grades than they deserved. If this audit is an accurate picture of how portfolios are being graded, then the problems with CATS are even worse that some of us thought they were.”

Out of the 136 portfolios that were originally scored distinguished, only 36 were scored distinguished after the audit. Out of 740 portfolios that were originally scored proficient, only 387 scored proficient after the audit. “One of the justifications for doing this kind of testing under KERA in the first place was to avoid the ‘Lake Wobegone Effect’ of norm-referenced testing,” said Cothran, “which occurs when a majority of students score above average. But findings like this seem to indicate that the Lake Wobegone Effect is alive and well right here in Kentucky.”

The Family Foundation supports the use of portfolios for instructional rather than assessment purposes.

###

¹ “Kentucky Commonwealth Accountability Testing System 2005-2006 Writing Portfolio Audit Report,” Appendix J, p. 24.

Story on passage of CATS testing bill

I was quoted in today's Lexington Herald-Leader story on the Senate's approval of SB 1, which would replace the CATS test.

Press Release: Senate approves testing change

FOR IMMEDIATE RELEASE
March 7, 2008
Contact: Martin Cothran
Phone: 859-329-1919

“It’s time to put CATS to sleep.”
— Martin Cothran

Family group praises Senate vote to replace CATS

LEXINGTON, KY—“We are pleased to see that lawmakers are finally heeding years of calls for changing the state’s testing system,” said Martin Cothran, senior policy analyst for The Family Foundation. “The CATS test has had 10 years to prove itself—18 if you count its prior incarnation as KIRIS—and it has yet to do so. How many generations of Kentucky’s children are we going to hold hostage to the reputations of those who have tied their political fortunes to the success of KERA? It’s time to put CATS to sleep.”

Senate Bill 1 was passed by the Kentucky State Senate in a 22-15 vote. The bill would replace the CATS testing system with a more objective, easier-to-administer-and-grade multiple choice test that would give reliable scores for individual students. CATS currently includes “open response” questions and portfolios that have been criticized as subjective and unreliable.

“This bill would give our testing system five things it doesn’t have now,” said Cothran. “It would be objective, easy to administer, easy to grade, reliable on an individual level, and would give us quicker feedback on how our students—and schools—are doing.”

Cothran was one of the most vocal critics of the testing system, and his criticisms led to some of the changes in KIRIS that resulted in the CATS test. He also served on the Assessment and Accountability Subcommittee of the Governor’s Task on Education Reform under former Gov. Paul Patton. He was the author of the minority report for the subcommittee.

###

Prichard Committee touting flawed critique of SB 1

Georgetown College's Center for Advanced Study of Assessment (CASA), it turns out, is not so expert after all. In a recent report being touted by the Prichard Committee, Skip Kifer, Ben Oldham and Tom Guskey criticized Senate Bill 1, which would replace the CATS test with tests that are actually objective, reliable, and useful. But, as it turns out, according to another testing expert, several of their criticisms got basic things wrong about the CATS test, calling into question the report's credibility.

George Cunningham, an emeritus professor from the University of Louisville, the author of numerous books on educational testing and a nationally recognized measurement expert, points out that the CASA report made fundamental errors in describing the CATS tests and what SB1 would do.

Is CATS a "standards-based" test?
The CASA report made the assertion that the CATS test was "criterion-referenced" or "standards based", and that SB1 would replaced it with a "norm-referenced" test:

The new legislation, while not requiring an off-the-shelf set of tests, appears to favor such an approach by requiring norm-referenced tests for individual students rather than the criterion-referenced or standards-based ones which historically the Commonwealth has used to measure school outcomes. (p. 7)

"The authors are confused," says Cunningham, "or perhaps just dated in their use of measurement terminology." "The criticism of SB1 tests that they will be norm-referenced is nonsensical because the current test, CATS, is also norm-referenced."

Oops. It might be a good idea, folks, before we start defending the CATS test to know what kind of test it is.

He points out that to say that CATS is somehow "standards-based" is misleading, and that it is only standards based in the same sense that all test are standards based:

The term “criterion-referenced” has lost its meaning. At one time it referred to the process of reporting results on an objective-by-objective basis and it was closely associated with mastery learning. Outside of special education, it would be difficult to find examples of this sort of criterion-referenced testing. Certainly, neither KIRIS nor CATS was ever criterion-referenced in this sense. Because the term apparently focus-groups well, a more modern usage of the term has emerged.

Ouch.

A "criterion-referenced" test is one that sets forth certain objective criteria and the score depends upon how a student meets those criteria. If a student, say, gets 6 out of 10 questions right, and 60 percent is a D on a predetermined grading scale, then the student gets a "D". A "norm-referenced" test is like test graded on a curve. If a student gets the same 6 out of 10, but the average in the class is a 6 out of 10, then the student gets a "C".

Cunningham's point is that neither the the KIRIS (the CATS before 1998) or CATS tests (KIRIS after 1998)--or the tests proposed by SB1 are "criterion-referenced". They're all norm-referenced. Of course Bob Sexton and the Prichard Commitee have been spreading this disinformation for years despite the fact that it has been pointed out publicly a number of times. In fact, I pointed it out in an opinion piece in the Herald Leader after the CATS test was first implemented.

Can multiple-choice tests measure complex knowledge and skills?
The CASA report repeats the completely unfounded assertion that multiple choice tests have some problem measuring advanced knowledge and skills:

The major strength of multiple-choice items in an assessment is that they are efficient. That is, in a relatively short amount of time, it is possible to get information about array of knowledge and skills. Their strength is not in measuring complex skills and knowledge.

Wrong again, Cunningham points out. "High quality, reliable and valid, off-the-shelf, standardized achievement tests are available to assess reading and math," he says, "...These available tests also do a good job of assessing high level thinking skills." In fact, Cunningham apparently considers the error bad enough to call CASA's credentials into question:

It is a little surprising to read a statement like this written by members of an organization that claims to focus on the advanced study of assessment. A more nuanced discussion about test type and high level thinking might be expected...It is axiomatic in educational measurement, that high level thinking is measured well by multiple-choice items. The authors should know this.

That's about as strong as academic take downs get. Once again, multiple choice tests can accurately and reliably measure high level thinking skills. In fact, it's done all the time. Just repeating a discredited view that they can't doesn't make it true.

I should point out here that I have questions concerning how well writing skills can be assessed using any system of measurement. Only another competent writer can assess competent writing. But that is not what is at issue here.

Are multiple choice tests less reliable for assessing schools?
The CASA report argues that the CATS test is a better measure of school performance than the more objective tests proposed by SB 1:

SB 1 changes the fundamental purpose of the assessment from emphasizing school outcomes to measuring individual student achievements. This, of course, has consequences. The most important one is whether the new emphasis and assessment is a better measure what Kentucky wants its schools to do ... The assessment envisaged by SB 1 would take, by design, a substantially narrower sample of the domain of desirable outcomes. (p. 8)

Well, not so fast. Says Cunningham, "There is no reason that test scores cannot be valid for both individual students and schools. Actually, the validity of school scores is dependent on the validity of individual students."

Kifer, Oldham, and Gusky acknowledge that matrix sampling renders individual students scores unusable but they claim that they make the school scores better. They assert that the SB 1 test sacrifices the validity of the school scores to get individual scores. While it is true that it is possible to include more open-ended items if multiple forms are used, by using a multiple-choice format even more items can be included, more than enough to compensate for the broader coverage from matrix sampling.

One wonders if the Prichard Committee had a role in getting this self-serving report produced in the first place, or whether they were attracted by the misinformative nature of it after the fact, and simply saw another opportunity to serve up disinformation. We do know that Helen Mountjoy, the Governor's education secretary requested the report, and that Mountjoy has long been a blind apologist for the state's flawed testing system. She has worked hand in glove with the Prichard Committee to oppose attempts to address the flaws in the tests. In any case, one wonders why there are those who still consider these people reliable sources of information.

Time to Put CATS to Sleep

Richard Day over at Kentucky School News and Comments offers coverage of Gov. Steve Beshear's press conference today attacking Senate Bill 1 , which would replace the controversial CATS test, saying, "The governor said the proposal has multiple flaws, and called on lawmakers to reject it."

Uh, wait a minute. Isn't that the exact argument being used against the CATS tests in the first place?

Bad choice of words, no doubt. But it does point up the incredible double standard going on here. Why are flaws in a bill an argument against the bill, but flaws in CATS--which have been pointed out, documented, argued over, fussed about, bemoaned, and, of course, swept under the rug--are not considered an argument against CATS?

Let's just cover briefly several qualities a test should have that CATS doesn't have:

• Objectivity
• Accuracy on an individual student level
• Reliability
• Ability to receive scores back in a reasonable amount of time
• Promotes basic skills

Now if you were told that a test you were considering didn't have these qualities, what in the world would possess you to use it? And how could you justify spending millions of dollars and countless man hours on the part of teachers and administrators to administer it?

Despite having no good answer to this question, we are still spending way too much money on the test, and there are still people willing to risk their credibility to defend it.

Go figure.