Gov. Sam Brownback threw out some statistics in his State of the State address Tuesday night that might have caught some people by surprise.

Among them was this: “29 percent of Kansas fourth-graders can’t read at a basic level.”

That figure is based on a somewhat controversial premise: the 2011 National Assessment of Educational Progress, or NAEP exam.

We’ve written about this before, but it’s probably worth mentioning again.

First, NAEP is administered by the U.S. Department of Education for the purpose of comparing states, and the nation as a whole, when large groups of students take the same test.

It is not administered to all students. Instead, it’s given to a “stratified random sample” of students in schools that, combined, reflect the state and national populations as a whole. In other words, you’re looking at survey results, not the results of a universal exam.

Second, the sole purpose of NAEP is to show how students in one state compare to other states, or the nation as a whole. It is very specifically not intended to be used as a measure of how any given student measures up against an objective standard of what kids should know at certain grades, which is what the governor attempted to do in his speech.

In education jargon, it’s called a “norm-referenced test,” as opposed to a “criterion-referenced exam.”

If you look at the 2011 NAEP scores for their intended purpose — to see how Kansas students stack up against students from other states who all took the same test — you would find that Kansas ranked 10th in the nation for the percent of fourth-grade students who scored proficient or better on the 2011 reading test.

If you look at just the average raw scores on the test, you would see that Kansas ranked 14th in the nation.

How can that be, you ask? How is it that a state where more than one in four students appear to be functionally illiterate nonetheless rank among the highest-performing states in the nation? Can the rest of the nation really be that dumb?

No, our state is not that dumb, and neither is the rest of our nation. The explanation lies in the fact that NAEP is a norm-referenced test, not a criterion-referenced test. In other words, the questions on the exam are not connected in any way with the educational standards adopted in this, or any other state.

NAEP administers a test. The scores are distributed across an array that can be plotted on a normal bell curve. Most scores are concentrated in the middle; smaller numbers of scores are scattered at the high and low extremes. The people who administer NAEP draw lines on the bell curve. They label one line “basic;” another line “proficient;” and another line “advanced.”

Those words, however, bear no relationship to any given set of standards. For that reason, the lines might just as easily be called the red, green and blue lines, or the Tom, Dick and Harry lines.

If you want to know how Kansas fourth-graders measure up to an objective standard, look at the Kansas state assessments. There, you would see that in 2011, 11 percent of Kansas fourth-graders (not 29 percent) fell short of the state’s established educational standards. In 2012, that number grew to 11.9 percent.

During his campaign in 2010, Brownback promised to improve fourth-grade reading scores in his first term. Tuesday night, the public got its first look at his plan for accomplishing that: ridding the fourth-grade class of any student who had trouble passing the third grade reading test the year before.

Third-grade retention laws like the one Brownback is proposing have been passed in a number of states, and it’s been debated in Kansas before but hasn’t yet gained enough support to get through the Legislature. It’s a serious issue that deserves serious study and discussion.

There’s a rule of thumb about teaching reading: Before third grade, students learn to read; after that, they read
to learn. It is critical that students have basic reading skills after third grade because from fourth grade on, they’re expected to be able to read their science, history and English textbooks in order to learn the material in them. (I deliberately left math textbooks out of that list because I’m still convinced that the first understandable math textbook hasn’t been written yet.)

Supporters of third grade retention laws argue that schools, and the state, are doing no favors to pupils by advancing them to fourth grade when they’re not ready. Without the ability to read with comprehension, supporters say, we’re just setting kids up for long-term problems that become harder to solve each successive year.

Critics of such laws argue that putting such a high-stakes test on he backs of third-graders is both unfair and cruel because they’re ill-equipped to cope with the pressure of taking a single test that can determine the entire course of their lives. Making a boy or girl repeat the entire third grade because he or she failed to ace a single standardized test may, in fact, do more harm than good because separating them from their peers of the same age and putting them in a class where all the other students are younger may serve to alienate them and increase the chance they will drop out of school later.

Supporters would counter that students who continually fall behind in school because of poor reading skills also are at a greater risk of dropping out.

Meanwhile, people on both sides seem to agree on at least one thing: neither holding a student back nor allowing “social promotions” will, by itself, solve his or her reading problem. Those students need more support and intensive work to help solve whatever problem they’re having.

The governor is proposing an additional $12 million a year, “to support innovative programs to help struggling readers,” an amount that may or may not be enough to fund intervention programs for dyslexia and a variety of other disabilities that may be at the root of a student’s problem.