A tedious method to scoring writing
By STEPHEN HEGARTY
© St. Petersburg Times, published May 2, 2001
JACKSONVILLE -- Ignoring the "Quiet Please" signs posted on partitions everywhere, Chris Citro stands near the center of a wide open conference room and calls for attention.
He is talking to the readers hired to score FCAT essays written by Florida's fourth-graders. Amid the hours of silent reading, Citro calls for these group "calibrating sessions" a couple of times each day to keep everyone sharp and head off potential problems.
A reader clutching a Dasani water bottle raises her hand.
"What if part of the essay is X'd out and you could still read it?" she says. Then she turns, as if explaining to her fellow trained readers. "It was clearly the best part of the paper."
Some poor kid somewhere in Florida wrote a terrific paragraph or two, then thought better of it and scratched it out. Now a woman in a conference room in Jacksonville must judge the fourth-grader's writing. Does she read it all? Or is that child's best bit of writing lost?
Here in an office park east of downtown Jacksonville, 250 readers have been slogging through 190,000 papers written by Florida's fourth-graders. (The eighth- and 10th-grade papers were scored elsewhere. The math and reading tests were scored later.) It's Citro's job to see that each reader looks for the same things. The woman holding the bottled water must be just as tough a reader as the guy nearby in the Bruins hockey jersey.
Throughout March and April, this scoring center and others like it were at the center of Florida's school accountability universe. This is where key parts of the FCAT (the handwritten stuff, not the fill-in-the-bubble answers tabulated by machine) are read and scored. This is where a couple of hundred college graduates were hired and trained to become human scoring machines.
Soon after the last paper is read and scored, and the tiny "NCS Pearson" sign is removed from the glass door in front, FCAT scores will be released to much fanfare. (They are expected this month).
Some schools will celebrate the scores and stand to earn tens of thousands of dollars in reward money. Others won't do well at all. They will be in danger of staff shake-ups and losing students to vouchers and private schools.
All that, based in part on what goes on in a sleepy office park east of downtown Jacksonville.
Mary Waterlander always wondered how they scored a writing test like this. In fact, she was skeptical about how they could score a test like this.
She's a former teacher, so her students took the FCAT. And she's a mother of three, so her own kids have taken it.
Now she's one of the scorers.
The Jacksonville woman saw an ad in the local paper. It called for "professional scorers" and it required a four-year college degree. The ad said nothing about the FCAT. But it sounded interesting.
Each reader underwent two or three days of training just to learn to read one type of fourth-grade writing exam. If they could accurately score seven or eight papers out of 10, they were hired. A dozen or so could not meet the accuracy standard. They weren't hired.
"We need people, but we really need people who are accurate," said Cornelia Orr, the Department of Education program director overseeing the test scoring.
Those hired won the right to earn $11 an hour for sitting in a conference room day after day throughout March reading essays written by fourth-graders. (The pay varies among the four scoring sites in Jacksonville, Minneapolis, Lansing, Mich., and Iowa City, Iowa.) Some of them were rehired to come back in April to be retrained and score short essays on the fourth-grade reading test.
For Waterlander and the other readers handling the writing test, the routine works like this:
They each get a blue packet of 20 handwritten papers. No student names or school names, only a five-digit code to identify students. Scorers read each paper, then assign each a score from one to six. They use the same scoring guide -- called a rubric -- that teachers and students use in Florida's classrooms.
With 190,000 papers to read, the scorers might get through almost 100 essays a day.
Later, each paper is read and scored by another trained reader. If the two scores are the same, that's the grade. If they're close -- one awards a three, the other awards a four -- the grade is the average of 3.5. If they are far apart, that paper goes to a third reader.
This is a far cry from the simplicity and speed of machine scoring multiple choice tests. These papers aren't scored in a mechanical blur. Here, the work is slow and deliberate. And Florida's standards -- the requirements for qualifying as a reader, and the procedure of having two, or even three, scorers read each paper -- adds to the time and cost of scoring.
One of the two "prompts" this year instructed the fourth-graders to write about something they made or built.
What did the kids write about?
"Lots of Legos. Lots of treehouses," said Rob Sights, the scoring director who oversees the narrative scoring this year. "Some kids get very creative, but believe me, you see a lot of the same things."
If Florida's teachers drill into their students a formulaic approach to the writing test (and many do), the trained readers suffer through the same five-paragraph essay, the same transitions, and the same sing-songy sentence structure over and over, day after day.
It's up to Sights and Citro and others who work for NCS Pearson to keep the trained readers sharp and consistent, despite the monotony. Every so often, a paper stands out. But when you read nearly 100 papers a day, you get all the five-paragraph essays on Legos and treehouses you can stand.
"We don't count that (formula) against them," Sights said. "And we don't penalize anyone for doing something different."
During his early afternoon "calibrating session," Citro talks about just how perfect the sixes must be. That's the highest score a paper can get. But, Citro explains, they don't have to be fully realized works of art.
"Even in a "six,' not everything is elaborated," says Citro. "In these sixes you could even chop off the last lines, and it's still a six."
Then comes the question of the fourth-grader who crossed out some of his best writing. Obviously the child didn't want those paragraphs read; he drew bold X's through it. But the scorer could read it. And it was quite good.
Does it count?
"If we can read it, read it," Citro says. "Especially if it helps the paper."
There are nods and a sense of relief around the room.
"Remember," Citro says, "the whole point is to see how well they can write."
© 2006 • All Rights Reserved • Tampa Bay Times
490 First Avenue South St. Petersburg, FL 33701 727-893-8111
From the Times state desk
From the state wire