St. Petersburg Times Online: News of Florida
Place an Ad Calendars Classified Forums Sports Weather
  • Fasano's firm eyed state job
  • Counties may help pay for vote reforms
  • A tedious method to scoring writing
  • Resentencing ordered on eve of execution
  • Senate okays grandparents rights
  • Law shields gunmakers from suits
  • Senators back off move to give felons the vote

  • From the state wire

  • Hurricane Jeanne appears on track to hit Florida's east coast
  • Rumor mill working overtime after Florida hurricanes
  • Developments associated with Hurricanes Ivan and Jeanne
  • Four killed in Panhandle plane crash were on Ivan charity mission
  • Hurricane Frances caused estimated $4.4 billion in insured damage
  • Disabled want more handicapped-accessible voting machines
  • USF forces administrators to resign over test score changes
  • Man's death at Universal Studios ruled accidental
  • State child welfare workers in Miami fail to do background checks
  • Hurricane Jeanne heads toward southeast U.S. coast
  • Hurricane Jeanne spurs more anxiety for storm-weary Floridians
  • Mistrial declared in case where teen was target of racial "joke"
  • Panhandle utility wants sewer plant moved to higher ground
  • State employee arrested on theft, bribery charges
  • Homestead house fire kills four children, one adult
  • Pierson leader tries to cut off relief to local fern cutters
  • Florida's high court rules Terri's law unconstitutional
  • Jacksonville students punished for putting stripper pole in dorm
  • FEMA handling nearly 600,000 applications for help
  • Man who killed wife, niece, self also killed mother in 1971
  • Producer sues city over lead ball fired by Miami police
  • Tourism suffers across Florida after pummeling by hurricanes
  • Key dates in the life of Terri Schiavo
  • An excerpt from the unanimous ruling in the Schiavo case
  • Four confirmed dead after small plane crash in Panhandle
  • Correction: Disney-Cruise Line story

    printer version

    A tedious method to scoring writing

    Since machines can't judge a students' writing ability, at least two people read each essay.


    © St. Petersburg Times, published May 2, 2001

    JACKSONVILLE -- Ignoring the "Quiet Please" signs posted on partitions everywhere, Chris Citro stands near the center of a wide open conference room and calls for attention.

    He is talking to the readers hired to score FCAT essays written by Florida's fourth-graders. Amid the hours of silent reading, Citro calls for these group "calibrating sessions" a couple of times each day to keep everyone sharp and head off potential problems.

    A reader clutching a Dasani water bottle raises her hand.

    "What if part of the essay is X'd out and you could still read it?" she says. Then she turns, as if explaining to her fellow trained readers. "It was clearly the best part of the paper."

    Citro winces.

    Some poor kid somewhere in Florida wrote a terrific paragraph or two, then thought better of it and scratched it out. Now a woman in a conference room in Jacksonville must judge the fourth-grader's writing. Does she read it all? Or is that child's best bit of writing lost?

    Here in an office park east of downtown Jacksonville, 250 readers have been slogging through 190,000 papers written by Florida's fourth-graders. (The eighth- and 10th-grade papers were scored elsewhere. The math and reading tests were scored later.) It's Citro's job to see that each reader looks for the same things. The woman holding the bottled water must be just as tough a reader as the guy nearby in the Bruins hockey jersey.

    Throughout March and April, this scoring center and others like it were at the center of Florida's school accountability universe. This is where key parts of the FCAT (the handwritten stuff, not the fill-in-the-bubble answers tabulated by machine) are read and scored. This is where a couple of hundred college graduates were hired and trained to become human scoring machines.

    Soon after the last paper is read and scored, and the tiny "NCS Pearson" sign is removed from the glass door in front, FCAT scores will be released to much fanfare. (They are expected this month).

    Some schools will celebrate the scores and stand to earn tens of thousands of dollars in reward money. Others won't do well at all. They will be in danger of staff shake-ups and losing students to vouchers and private schools.

    All that, based in part on what goes on in a sleepy office park east of downtown Jacksonville.

    * * *

    Mary Waterlander always wondered how they scored a writing test like this. In fact, she was skeptical about how they could score a test like this.

    She's a former teacher, so her students took the FCAT. And she's a mother of three, so her own kids have taken it.

    Now she's one of the scorers.

    The Jacksonville woman saw an ad in the local paper. It called for "professional scorers" and it required a four-year college degree. The ad said nothing about the FCAT. But it sounded interesting.

    Each reader underwent two or three days of training just to learn to read one type of fourth-grade writing exam. If they could accurately score seven or eight papers out of 10, they were hired. A dozen or so could not meet the accuracy standard. They weren't hired.

    "We need people, but we really need people who are accurate," said Cornelia Orr, the Department of Education program director overseeing the test scoring.

    Those hired won the right to earn $11 an hour for sitting in a conference room day after day throughout March reading essays written by fourth-graders. (The pay varies among the four scoring sites in Jacksonville, Minneapolis, Lansing, Mich., and Iowa City, Iowa.) Some of them were rehired to come back in April to be retrained and score short essays on the fourth-grade reading test.

    For Waterlander and the other readers handling the writing test, the routine works like this:

    They each get a blue packet of 20 handwritten papers. No student names or school names, only a five-digit code to identify students. Scorers read each paper, then assign each a score from one to six. They use the same scoring guide -- called a rubric -- that teachers and students use in Florida's classrooms.

    With 190,000 papers to read, the scorers might get through almost 100 essays a day.

    Later, each paper is read and scored by another trained reader. If the two scores are the same, that's the grade. If they're close -- one awards a three, the other awards a four -- the grade is the average of 3.5. If they are far apart, that paper goes to a third reader.

    This is a far cry from the simplicity and speed of machine scoring multiple choice tests. These papers aren't scored in a mechanical blur. Here, the work is slow and deliberate. And Florida's standards -- the requirements for qualifying as a reader, and the procedure of having two, or even three, scorers read each paper -- adds to the time and cost of scoring.

    One of the two "prompts" this year instructed the fourth-graders to write about something they made or built.

    What did the kids write about?

    "Lots of Legos. Lots of treehouses," said Rob Sights, the scoring director who oversees the narrative scoring this year. "Some kids get very creative, but believe me, you see a lot of the same things."

    If Florida's teachers drill into their students a formulaic approach to the writing test (and many do), the trained readers suffer through the same five-paragraph essay, the same transitions, and the same sing-songy sentence structure over and over, day after day.

    It's up to Sights and Citro and others who work for NCS Pearson to keep the trained readers sharp and consistent, despite the monotony. Every so often, a paper stands out. But when you read nearly 100 papers a day, you get all the five-paragraph essays on Legos and treehouses you can stand.

    "We don't count that (formula) against them," Sights said. "And we don't penalize anyone for doing something different."

    * * *

    During his early afternoon "calibrating session," Citro talks about just how perfect the sixes must be. That's the highest score a paper can get. But, Citro explains, they don't have to be fully realized works of art.

    "Even in a "six,' not everything is elaborated," says Citro. "In these sixes you could even chop off the last lines, and it's still a six."

    Then comes the question of the fourth-grader who crossed out some of his best writing. Obviously the child didn't want those paragraphs read; he drew bold X's through it. But the scorer could read it. And it was quite good.

    Does it count?

    "If we can read it, read it," Citro says. "Especially if it helps the paper."

    There are nods and a sense of relief around the room.

    "Remember," Citro says, "the whole point is to see how well they can write."

    Recent coverage

    New scoring for FCAT may add to grade gap (April 12, 2001)

    Diploma to hinge on FCAT success (March 7, 2001)

    Crist works late to make tough FCAT essays count (March 6, 2001)

    Psst: Some test questions don't count (February 11, 2001)

    Patience tested by FCAT delays (May 28, 2000)

    Back to State news
    Back to Top

    © 2006 • All Rights Reserved • Tampa Bay Times
    490 First Avenue South • St. Petersburg, FL 33701 • 727-893-8111
    Special Links
    Lucy Morgan

    From the Times state desk