Front Page
  Current Issue
  Back Issues
  About National CrossTalk

News Editorial Other Voices Interview

2 of 5 Stories

A Contrarian View of the Testing Industry
FairTest argues that standardized tests are a poor predictor of student success

By Robert A. Jones
Cambridge, Massachusetts

Monty Neill, the executive director of FairTest, stood in an empty room on an empty floor of a vintage office building near Harvard University. He motioned to a corner. That's where a FairTest researcher once worked. And over there, along the wall, stood a row of file cabinets packed with research materials going back 20 years.

Robert Schaeffer, one of FairTest's founders and still its public education director, stood next to Neill, looking a bit uneasy. Even now, he said, it can feel embarrassing to talk publicly about the near collapse of the organization that has consumed most of his life's work. FairTest, he said, has played its role as gadfly in the world of standardized testing for so long that many assumed it could not stumble and fall.

But stumble it certainly did. Over the past two years FairTest has progressively retrenched as its financial backers, mostly foundations, withdrew their support. Last October, the situation became so dire that the Board of Directors briefly considered shutting its doors. Ultimately they decided to hunker down instead, leaving only Neill and Schaeffer on the payroll and shrinking the office space to a fraction of its former size.

Now, nine months later, FairTest's crisis has eased somewhat. Revenues from the organization's website increased in the spring, thanks largely to the group's role in uncovering the SAT scoring scandal at FairTest's longtime adversary, the College Entrance Examination Board. Visits to the website increased dramatically along with some private donations.

Monty Neill is executive director of FairTest, persistent critics of standardized testing, especially the SAT.
(Photo by MJ Maloney, Black Star, for CrossTalk)
The notion of FairTest being saved by the errors of the College Board may seem ironic but, in fact, it is emblematic of the group's history. From its start in 1986, FairTest has played the role of outsider in the clubby, often opaque world of standardized testing. It has specialized in pricking the reputations of the College Board and other institutions, arguing that their much-feared tests are often faulty by design and, more often than not, fail in their primary mission of predicting student success.

Upstairs, in the remaining office, Schaeffer said the reprieve has given FairTest the chance to continue offering its contrarian services. "There's always been certain groups who have been attracted to us because we offer the other side of the story," he said. "If you want a non-industry, non-establishment view of the testing industry, we can provide it. Frankly, there's no one else who can do that."

Over the years FairTest and its allies have made notable gains in their struggle with the industry. The testing process now is more open, and industry research is available to the public. Test questions contain less bias. And the SAT and its midwestern rival, the ACT, have been stripped of some of their fearful power by the acknowledgement of test makers that test scores are not immutable but can be altered through coaching.

Throughout, FairTest has proved difficult to ignore because of its aggressiveness and in-your-face style. When the group announced its founding in 1986, for example, it did so at the College Board's biggest event of the year, the College Board Forum, where it proceeded to lambaste the SAT for various alleged biases against minorities and low-income groups.

And just recently, when Schaeffer appeared opposite Gasper Caperton, president of the College Board, at a New York state hearing on the SAT scoring scandal, Schaeffer told legislators that the breakdown in scoring proceeded from the fact that the testing industry "has no enforceable quality control standards and lacks basic accountability to students, teachers and the public.

"The truth is, there is stronger public oversight and control over the food we feed our pets than for the tests administered to our children."

This approach, essentially political, does not mesh easily with the academic style of the psychometricians and administrators of the testing industry, and it has not won FairTest many fans within that world. Several prominent members of the industry, in fact, refused to comment on FairTest for this story.

One such leader, Kurt M. Landgraf, president of the Educational Testing Service, not only declined to comment himself but had an ETS spokesman convey the message that the entire organization would remain mute on the subject.

Wayne Camara, vice president for research at the College Board, was one of the few willing to discuss the group. After expressing his support for several of FairTest's positions on standardized testing, Camara then ripped into their brash approach, particularly the recent comment comparing the SAT to pet food.

Laura Barrett, FairTest's board chairman, thinks college applicants should be judged on their overall performance, not just on test scores.
(Photo by MJ Maloney, Black Star, for CrossTalk)
"That's a great sound bite," he said. "But what does it mean? It's not like we're manufacturing pet food, where you open up a can and test it for the nutrients. I mean, it's a spurious issue. What regulatory body was going to catch the scoring error? Would this (regulatory) agency watch them scoring the test, overseeing what they are doing?"

In actuality, FairTest's approach is more or less the classic strategy of underdogs in the world of public policy. Snappy language and sound bites are used by small players to attract attention when they are hopelessly outgunned by larger rivals. With FairTest, the brashness has given the group an influence far larger than its tiny size and budget would suggest.

That influence extends to college admissions officers and high school guidance counselors around the country where FairTest's arguments are widely known and often supported. In many such offices the decades-long willingness to accept SAT or ACT scores as holy writ has been broken, probably forever. And in its place, a more complex system of assessing students has arisen.

The most vivid evidence of this change can be seen on the FairTest website (, which now lists more than 730 colleges and universities around the country that allow some or all applicants to forgo submitting test scores for admission. Some institutions allow all students to apply without test scores, and base their review on high school grades, portfolios of student work, personal interviews and other material. Other colleges allow test scores to be omitted only if students have achieved grade point averages of a certain level or higher.

The 730 institutions on the list include a wide spectrum ranging from state universities to elite private schools. No Ivy League schools have gone test-optional, but, according to FairTest, the list now includes 25 of the top 100 liberal arts colleges in the U.S. News and World Report rankings.

Actually, FairTest did not originate the test-optional movement. That role falls to Bates College in Maine, which took its action a year before FairTest was founded. But FairTest has tirelessly promoted the idea that students and colleges are better served when admissions officers employ an array of assessment measures rather than relying on test scores as a principal guide.

Every year, more colleges and universities make SAT and ACT scores optional for some or all applicants, says Robert Schaeffer, FairTest's national education director.
(Photo by Todd Anderson, Black Star, for CrossTalk)
"FairTest has acted as a bully pulpit for test-optional," said William Hiss, Bates' vice president for external affairs, who served as its admissions officer when the college made its original decision. "They have also acted as a visible presence for a broad movement that questions the values of standardized tests and the way they are used in America. I think that amounts to a valuable service."

While Bates and FairTest both say they have kept an arms-length relationship, the Bates experience nonetheless has proven invaluable to FairTest's argument that standardized tests have dubious value in predicting the success of college students.

The results of the Bates program have been documented over the 21 years of its existence, and some of the results have proved startling. Bates has found, for example, that students who did not submit test scores, known as "non-submitters," maintained GPAs that were virtually the same as submitters. The graduation rates of non-submitters were also virtually the same as submitters.

Those results are all the more salient because Bates is a highly selective college, and test-submitting freshman have high average test scores. So the non-submitters at Bates compete with high-end SAT scorers, and differences in outcomes still are hard to detect.

Hiss said Bates' experience has converted him and other college officials to the theory of multiple intelligences advanced by Harvard educator Howard Gardner. No single test can measure the different forms of intelligence, talents and skills in young people, Hiss said, and such a test inevitably will cut out some promising students whose skills do not appear on the test results. Such a process, he said, raises grave questions for colleges and the country as a whole.

For one thing, he argues that rejecting students who would otherwise thrive in college can have crippling impacts on their lives, possibly pushing them from college altogether and leaving them with career choices that are far below their real talents.

And further, he said, the whole nation suffers. "When you use a test that artificially reduces the pool of people who could go to college and succeed, you are truncating the number of educated people available for higher-level jobs. That is terrible economic policy," said Hiss.

In Cambridge, Schaeffer says FairTest now receives several phone calls a month from one college or another asking about the process of going test-optional. In the last three years the list has grown by more than 200 schools.

"We expect to see the growth (of the list) accelerate," said Schaeffer. "In fact, the question now gets raised as to whether we are reaching a tipping point where the numbers start to mushroom."

The implication of the test-optional movement—that the SAT and ACT are expendable—has clearly hit a sensitive nerve within the industry, and there has been at least one attempt to erode its significance.

In 2002, the College Board assigned two researchers to conduct a study of FairTest's list of test-optional schools which concluded that FairTest had skewed the list, exaggerating its size.

Educator Deborah Meier says FairTest can "keep the testing companies honest-there's no one else who does that."
(Photo by Peter Finger, Black Star, for CrossTalk)
At the time, the list contained 390 schools. The College Board claimed that 52 of the institutions on the list did not belong on the list because, in fact, they required admissions tests for all students. "Clearly, the FairTest misleading," the study said, and later concluded, "It is imperative that the number of SAT/ACT-optional institutions not be overestimated."

But FairTest officials say it was the College Board study, not their list, that contained the error. "There were no 52 colleges that didn't belong on the list," said Schaeffer. "We review the list all the time, and we are vigilant about pulling off any colleges that get placed there by mistake. I'm not saying they fabricated their list, because I don't know how or why the mistake was made, but their claim just wasn't true."

Schaeffer's complaint appears to have some justification. The names of the 52 institutions were not supplied by the study, but the College Board made them available to National Crosstalk after a request. A random spot check of five colleges on the list revealed that all five have test-optional programs in one form or another, and all five had them at the time of the study.

When FairTest rented its first office just off Harvard Square in 1986, its future impact on the education world would have been hard to predict. Among other things, it was the product of a political odd couple who shared almost nothing except a deep suspicion of standardized testing of all kinds.

John Weiss, a college drop-out and left-leaning political operative, had first become involved in the test reform movement in the late 1970s, when he worked for Massachusetts Congressman Michael Harrington. Those reform efforts eventually fizzled and Weiss repaired to Maine to work in community organizing.

Weiss was still in Maine when, one day, he was approached by a private detective. The detective told him he had been sent by a man named J. Patrick Rooney who wanted to talk about standardized test reform. The detective gave him Rooney's phone number.

Indianapolis businessman J. Patrick Rooney contributed several million dollars to FairTest in its early years.
(Photo by Clint Keller, Black Star, for CrossTalk)
"Rooney and I talked and at first I said I wasn't interested. I didn't want to start a new organization. I was burned out, that's why I went to Maine," said Weiss, who is now publisher of the Colorado Springs Independent, an alternative weekly, and a member of FairTest's board. "Then I got curious and asked how much money was available. Rooney said $750,000. Well, that was a lot of money back then. So I thought for a minute and said, maybe I am interested."

Rooney, it turned out, was the founder of the Golden Rule Insurance Co. of Indianapolis, a wealthy man and politically eccentric conservative who also had championed civil rights throughout his life. Rooney's company aimed many of its products at blacks in the midwest, and his interest in standardized testing was piqued when he discovered that white insurance salesmen were passing the licensing exam in Illinois in large numbers but black salesmen were not.

The exam infuriated Rooney. "In my judgement the (Illinois) testing was used for discrimination purposes," he said. "The construction of the questions had a disparate impact on minority people, and I believe it was intended."

Rooney sued Illinois and the Educational Testing Service, which provided the tests for the state. The result, which is still known as the "Golden Rule Settlement," committed the state and ETS to reform the way questions are formulated for its licensing tests.

When Rooney called Weiss, his objective was to expand the reforms he had won in the lawsuit to cover all types of standardized tests, from college admissions to K–12 and employment testing. It was an ambitious plan. At one early point, after Weiss had recruited Schaeffer to join him, the two constructed a back-of-the-envelope organization chart that showed they would need a staff of nine to advance their cause on all three fronts.

They never quite made it. Rooney contributed several million dollars to FairTest's operations over a five-year period, and he was followed by grants from the Ford, Joyce and McIntosh Foundations. The group hit its high-water mark in the late '90s when the staff grew to seven persons occupying two floors of the current office building.

Laura Barrett, FairTest's board chairman, said that, even then, the struggle with the large test makers was hardly a battle of equals. "On one side you have these very large institutions that make a lot of money from testing. And they spend it on promoting their products. On the other side you have this small organization that represents the constituency of people who have concerns with testing, but nobody makes money from that. So it's been tough."

Soon it would get tougher yet. By fall 2005 all three of the supporting foundations had dropped their support and FairTest was surviving on crumbs. Several members of the board made personal donations that allowed them to keep the doors open temporarily, and it was decided to give Neill and Schaeffer several months to generate some backing.

No one is certain why the foundation support dried up but Neill believes part of the reason lies with George Bush's No Child Left Behind legislation that mandates K–12 testing across the country.

"It's affected the foundations in the sense that they perceive there's a national consensus about No Child Left Behind and that more K–12 testing is inevitable," Neill said. "They see Ted Kennedy and George Bush in alliance on this, and they're asking themselves why they should fund this small group (FairTest) that's saying no, no, it's not going to work, it's bad news."

The FairTest leaders also acknowledge a more difficult truth: their group is well-known for what it stands against—standardized testing in most of its forms—but less well known for what it supports.

Diane Ravitch, education professor at New York University and senior fellow at the Hoover Institution, said she agrees that standardized tests have their issues but that FairTest has never met a test it liked. "In my experience, FairTest is opposed to all standardized tests," she wrote in an email. "I don't know of any that they think is 'fair.'"

This view, not rare in the education world, may also have hurt the group in terms of foundation funding. Foundations often prefer to fund organizations that pursue a benign course toward their goal. FairTest, in the words of one critic, may present to the foundations as "too edgy, too negative."

But, in fact, FairTest has long promoted its own view of the way student assessment should work. That approach—whether applied to college admissions or the K–12 arena—resembles the "portfolio" method used at some colleges, whereby students present a multi-layered profile of their high school careers both inside the school and out.

FairTest would have K–12 teachers evaluate students individually for their proficiency in various subjects, and the portfolios would include samples of essays, projects and other work that revealed their talents or deficiencies.

Though any individual evaluation would involve some subjectivity, Neill says, the evaluation of an entire school of students under this system would prove far more reliable and revealing than the single multiple-choice test now used by many school districts.

Barrett, who once served as executive director of the group, says this type of student assessment has been tried successfully by a few districts but the effort has not drawn much attention. "It's hard to explain, it's complicated and you can't reduce it to a number," she said.

At least one member of the FairTest board argues that the rap about negativity is a red herring and the group should not apologize for playing the role of critic. Deborah Meier, a founder of progressive schools in New York and Boston and a former MacArthur Fellow, said, "I'd think that the foundations would want to keep FairTest around just to keep the testing companies honest. There's no one else who does that. The testing companies operate in a world of secrecy, and they have this power over people's lives. We need a counter-weight organization that opens up the process and tells us the other side of the story."

Bob Laird, former undergraduate admissions director at the University of California, Berkeley, says that FairTest's role as counter-weight also has been valuable in college admissions offices. "The College Board is very skilled at building a web of relationships with deans, admissions officers and college presidents," he said. "They are never so direct as to say, 'We are inviting you to a conference at the Toronto Four Seasons, and then we expect you to use our products,' but the connection nonetheless gets made.

"FairTest is valuable because they are addressing the other end of that equation. They face a Sisyphean task in many ways, but they are making gains."

And what of the future? Camara, the research vice president at the College Board, predicts that the movement to diminish the role of the SAT and ACT will be limited by the realities of the admissions process, especially at large schools.

Camara told the story of a visit he paid several years ago to the admissions office at UCLA. That year, he said, UCLA had approximately 4,500 spaces in its upcoming freshman class and had received more than 40,000 applications. Of those, about 10,000 applicants had high school GPAs of 4.0 or higher.

"So UCLA was looking at 2.5 applicants with 4.0s for every available spot," Camara said. "In a situation like that, how are you going to make the decision about who gets in? Do you flip a coin? Or do you use the SAT or ACT?"

Students who do not submit test scores do as well at Bates College as those who do, says William Hiss, vice president for external affairs.
Laird is not so sure that the tests would be the best way to solve that dilemma. When the UCLA story was described to him, he laughed. "At Berkeley we had years where our situation was more extreme than UCLA's. But I'd say, yes, you certainly could make the decision without the SAT.

"You look at each applicant, not just in terms of their numbers, but in terms of their range of accomplishments and the environment they come from. What course load did they take? What was their high school like? What kind of community did they come from? You can get a comprehensive picture of what the kid achieved versus what was available to him or her. That's what we did at Berkeley, and it worked extremely well."

For Laird, the growing perception of the limited usefulness of the SAT suggests that its greatest days might have passed. "I think the College Board is probably preparing for the day when the SAT is no longer its most profitable vehicle," he said.

As for the future of FairTest, prospects have improved, but a return to flush times is far from guaranteed. The group received several small grants this spring, and the influx of private donations means that the present, scaled-down operation can continue into next year.

But Neill and Schaeffer would like to see the organization return to its former size of seven or so staff members. "I feel like we've moved out of critical care and now we're ready for rehabilitation," said Schaeffer.

That will require a several-year commitment from a foundation or private patron. The group's request for funding is now being considered by several foundations, and decisions are expected within several months.

There is another issue that faces FairTest. Both Neill and Schaeffer have worked at testing reform for decades now. Their hair has turned gray during their tenure, and their years remaining at the helm likely are not long.

So who will pick up the cudgels when they have left?

Schaeffer laughs at the question. "We had a great pool of talented young people. The only problem is, we had to lay them off. Maybe we should try and find out where they went."

Robert A. Jones is a former reporter and columnist at the Los Angeles Times.

E-Mail this link to a friend.
Enter your friend's e-mail address:

National CrossTalk Summer 2006



National Center logo
© 2006 The National Center for Public Policy and Higher Education

HOME | about us | center news | reports & papers | national crosstalk | search | links | contact

site managed by NETView Communications