Testing the limits

President Bush remains adamant that every child in grades 3 to 8 must be tested as a way to gauge school improvement. Experts say that will place a huge strain on the companies that do the testing and the states that have to pay for it.

By STEPHEN HEGARTY

© St. Petersburg Times, published May 2, 2001

Last year, the company scoring Florida's FCAT test explained that the scores would be late because there weren't enough qualified people available to accurately grade the papers.

This year the company, NCS Pearson, is doing a better job meeting deadlines. But it wasn't easy hiring hundreds of college-educated people to read answer sheets day in and day out for modest pay.

So what happens if President Bush and Congress force all states to adopt multigrade testing programs?

"A train wreck -- that's what happens," said H.D. Hoover, president-elect of the National Council on Measurement in Education and the senior author of the Iowa Test of Basic Skills.

Hoover and others think that problems in Florida and elsewhere are evidence that the three big companies that handle the majority of tests nationwide are having trouble keeping up with demand -- and, if the president's education proposals are passed, the demand will go through the roof.

Bush's proposal would require testing 22-million students. The cost of testing grades 3 through 8 has been estimated at $7-billion over the next seven years, according to the National Association of State Boards of Education.

In an interview Tuesday with education reporters from the St. Petersburg Times and other newspapers, Bush said he thinks those cost estimates are inflated and that the job can be accomplished with the help of available federal dollars.

"You're going to hear reports saying, "Well, it costs billions to test,"' Bush said. "Whoever is trying to undermine the accountability system by putting out numbers that simply aren't realistic needs to hear the facts."

Still, testing companies expect the industry to feel the strain if Congress passes the program in its current form.

"There's going to be an explosion in testing," said Jennifer Vranek, director of benchmarking and state services for Achieve Inc. "The testing companies have already shown signs of strain. You've seen some of that in Florida."

Vranek's organization, a not-for-profit group formed by governors to help states implement school reform, estimates that 260 new tests will have to be created if all states answer Bush's call for reading and math exams in grades three through eight.

Forty-eight states require some sort of tests already, but only 15 have math and reading tests in all the grades that Bush would like. Florida, with tests in grades three through ten, is included among the 15.

"It's about filling in the gaps, but there are lots of gaps," Vranek said.

The nation seems to have a love-hate relationship with testing these days.

On one hand, colleges are questioning the role SAT scores have played in college admissions for generations. In some states, teachers and parents are protesting what they see as an overemphasis on testing in public schools. Despite all that, experts expect more tests, tougher tests, and higher stakes. Why? School accountability. Both at the federal and state level, lawmakers rely on tests to help them hold schools and teachers accountable.

As the president of the Educational Testing Service said in congressional testimony in March, it's difficult to know if reforms are working if testing isn't done to measure changes.

"Without standardized testing," said ETS president Kurt M. Landgraf, "parents and taxpayers can't know how much their students have learned relative to standards or to other students."

There are less noble justifications. As Hoover puts it, testing can enable states to appear to be serious about education.

"As expensive as testing is, it's less expensive than paying teachers more," Hoover said.

The challenge for testing companies is not just the increase in testing; it's the kind of testing that has caught on in recent years.

Companies have been machine scoring vast numbers of multiple choice, fill-in-the-bubble answer sheets with lightning speed -- and impressive accuracy -- for years. But now states are opting for more sophisticated tests that include open-ended questions that require written answers, short essays, or hand-drawn diagrams and graphs.

That kind of answer sheet cannot be scored by a machine. It requires educated people who have to be trained. And that requires time and money.

In Texas, the statewide test costs about $4 per student, according to a study done by Achieve Inc. In Maryland, it's closer to $30 per student. What's the difference? Texas' test is mostly multiple choice. Maryland's test relies more on extended answer items.

Florida's FCAT average cost estimates are somewhere in the middle. The machine-scored math and reading tests each cost $2 per student, while the performance items for math and reading each cost $11 per student. The writing test costs $5 per student.

As more states opt for the more sophisticated short answer or extended essay tests, the burden on testing companies increases. That's especially true for the companies that score tests, and the king of that niche is the company scoring Florida's FCAT, NCS Pearson of Iowa City, Iowa.

This year, NCS Pearson has opened 15 scoring sites, including Jacksonville, Iowa City and Lansing, Mich. NCS is hiring 150 to 300 scorers per site. That means they have had to hire and train 3,000 to 4,000 scorers nationally.

They opened more sites this year than last, which reduced the pressure of finding great numbers of qualified scorers in any particular city, according to Gary Mainor, president of assessments and testing for NCS Pearson.

The company chooses scoring sites based on several factors, not the least of which is demographics. The scoring site in Jacksonville drew a mix of military retirees and students from the nearby University of North Florida.

Experts see problems ahead.

"How many people in the world have a college degree and have this kind of time to read papers for what is essentially minimum wage?" Hoover asked.

What can be done?

In the future, scoring will be done by permanent trained scorers via computer. Forget the sites and the need to find a core of qualified readers in a given city. Technology opens up the possibility of full-time readers working at home. But that future is not yet here, due to security questions.

Mainor of NCS Pearson said there are simpler ways to ease the strain. If states gave tests at different times, it would be easier to score great numbers of tests. Mainor cautioned: Don't count on it.

"Current trends in state programs are not in our favor," Mainor said. "Test later and report before school is out' seems to be the prevailing desire."

That certainly is the prevailing desire in Florida.

After test results were late last year, then-Education Commissioner Tom Gallagher worked out an agreement with NCS Pearson to ensure test results are on time.

Gallagher agreed to reduce the number of extended answer items (six reading items and four math, compared with eight and six, respectively, last year), and to accept those results later -- thereby making the scoring easier. NCS agreed to give the test later and return the machine-scored multiple-choice results earlier.

Hoover said another simple solution would be to have trained readers score tests for more than one state. If the tests were more uniform, it would be possible for test scoring companies to get more out of trained scorers.

But states tend to create tests, as Florida did, specifically tailored to their students and their standards. The readers who score Texas' TAAS test can't be expected to shift gears and score the FCAT -- at least not without being trained again.

"This is not just an education issue, it's a political issue," said Hoover. "The people making the decisions are politicians. There are solutions, but I don't see us adopting them any time soon."

Recent coverage

New scoring for FCAT may add to grade gap (April 12, 2001)

Diploma to hinge on FCAT success (March 7, 2001)

Crist works late to make tough FCAT essays count (March 6, 2001)

Psst: Some test questions don't count (February 11, 2001)

Patience tested by FCAT delays (May 28, 2000)

© 2006 • All Rights Reserved • Tampa Bay Times
490 First Avenue South • St. Petersburg, FL 33701 • 727-893-8111

Contact the Times | Privacy Policy
Standard of Accuracy | Terms, Conditions & Copyright