Front Page
     
  Current Issue
     
  Back Issues
 
  Searchable
CrossTalk
Index
 
  Download
 
  Subscribe
 
  About National CrossTalk

National CrossTalk Fall 1999
News Editorial Other Voices Interview

2 of 5 Stories

Trying to Measure Student Learning
Missouri gathers a lot of data, but what does it all mean?

By William Trombley
Senior Editor

Kirksville, Missouri

Students at Truman State University, in this small town in the northeastern corner of Missouri, are tested, interviewed, surveyed and assessed within an inch of their lives. Truman is a "bastion of measurement," said a former campus administrator. "If it moves, they measure it."

Freshmen take a general education test during their first week on campus, then take the same test again as juniors, to measure their progress. Sophomores must successfully complete a "writing experience" which consists of writing an essay and then having it evaluated by a faculty member in a one-on-one session. Seniors are tested in their majors and also must present portfolios that are supposed to reflect their academic experiences at Truman. All of this is in addition to the tests they take, and grades they receive, routinely in college courses.

Students are surveyed or interviewed frequently while on campus and additional questionnaires pursue them after graduation, asking about jobs and other aspects of their lives as alumni. "I wonder if I'll have to fill out something when I die?" one student asked a recent campus visitor. "This is an institution that believes in assessment," said Robert Stein, associate commissioner for academic affairs at the Missouri Coordinating Board for Higher Education. "If you're serious about changing the campus culture, assessment has to penetrate throughout and it does at Truman." And not only at Truman.

Missouri is one of a handful of states with a comprehensive student assessment plan. Each of the state's 13 public four-year campuses, and most of its two-year colleges, test students in both general education and in academic majors or technical specialties. Some do so more grudgingly than others. The Coordinating Board for Higher Education uses these results, along with other factors, to determine how much money should be allocated to each campus from two incentive funds, one called "Funding for Results," the other "Mission Enhancement."

Missouri does not insist that a particular general education test be used, as a few other states do. However, to be eligible for the incentive money, a campus must use a nationally normed test, so the results can be compared with other states. Most campuses use either the CAAP (College Assessment of Academic Proficiency) test, produced by ACT, or the Educational Testing Service's "Academic Profile." Tests in academic majors include some that are nationally normed and some that are not.

Many faculty members and campus administrators regard the Missouri assessment program as little more than a public relations effort, designed to placate politicians who question whether the state is getting its money's worth. (This year, Missouri is spending about $1.2 billion on postsecondary education.) "In the long run, some good will probably come of all this but a lot of it is PR and some of it is smoke and mirrors," said an official at the University of Missouri's flagship campus in Columbia, where resistance to assessment has been strongest.

"This is part of the whole national movement toward accountability," said Deborah Carr, director of teacher development at the University of Missouri's education school. "It's beginning to trickle up from KĐ12 to higher education."

 
President Jack Magruder of Truman State University says student assessment “has been very good for this university.”  
In Missouri, student assessment had its beginnings in the mid-1970s at Truman State, then called Northeast Missouri State University. Charles J. McClain, president at the time, was not convinced that the university's graduates were as well-prepared as they should be.

"Everybody kept telling me what a great job we were doing but I thought, 'compared to what?'" McClain said in a recent interview. "I set out to get the data on how well we compared nationally and when the numbers came in, it was clear we weren't up to national standards," he said.

McClain and Darrell Krueger, his dean of instruction, quietly began to put in place a student assessment program that depended largely on nationally normed tests.

"There was no announcement by the president, there was no faculty meeting," McClain recalled. "We just started collecting data and sharing it with the departments. It was a very gentle process," although "it wasn't always easy to tell faculty members they weren't as good as they thought they were." The system was "pretty well in place" by the early 1980s, McClain said. "I think it changed the campus discussion to one about teaching and learning, and that was a wonderful thing."

The president had another motive. He knew that Northeast Missouri State would not flourish, and might not even survive, in a remote, bleak part of the state where population was declining steadily. McClain hoped that the assessment program, which was beginning to show results in terms of higher test scores and better retention and graduation rates, would gain some national attention for his isolated campus.

And it did. Northeast Missouri State's name began popping up in academic reports and journalistic articles as the "accountability" movement began to gather steam nationally.

"It was a marketing device, no two ways about it," said Peter Ewell, senior associate at the National Center for Higher Education Management Systems, who has written extensively about assessment. "Charles McClain did it quite deliberately and it worked." (Ewell is now a member of the governing board at Truman.)

Performance funding, based partly on student assessment, has had enormous appeal for the nation's governors and legislators, who spend billions of dollars each year on higher education but have only the dimmest notions about the results. Twenty states now tie postsecondary funding directly to performance, according to Joseph Burke, who has been studying the issue for several years as director of the higher education program at the Rockefeller Institute, in Albany, N.Y.

 
  Missouri Commissioner of Higher Education Kala Stroup believes student assessment is now well established in the state.
Missouri began to explore performance funding in the late 1980s, when Republican Governor John Ashcroft (now U.S. Attorney General) and a group of influential business and education leaders began to promote the idea. At first, there was considerable resistance.

"This was a very tough ship to turn around," said Sandra Kauffman of Kansas City, who was a member of the state House of Representatives from 1987 to 1998. "There was a real reluctance on the part of the administrative family to be questioned, to provide data to show that they were doing their job." A 1991 report by the Business and Education Partnership brought change, Kauffman said. "The business community made it clear what their expectations were for postsecondary education in Missouri and that they just weren't being met."

Although Ashcroft supported performance funding, he provided no state funding for this purpose. But his successor, Democrat Mel Carnahan, did, including $3 million in his first state budget. Robert Stein, of the Coordinating Board for Higher Education, called this a "small but symbolic commitment."

This was the beginning of "Funding for Results" (FFR), which has generated more than $66 million for the state's public colleges and universities since 1994. Public campuses have received another $137.5 million from the second incentive program, "Mission Enhancement," which began in 1997. These funds become part of the ongoing campus base budget; they are not a one-time bonus.

In allocating the FFR dollars, the coordinating board considers not only the results of student assessment but also graduation and retention rates; performance on national exit exams in such fields as teaching and nursing; and the number of minority students receiving degrees or certificates, among many other factors.

Some campuses work hard to earn the incentive money, others do not, but coordinating board officials believe the statewide effort has been generally successful.

"Assessment and this whole process are what people talk about now," said Kala Stroup, Missouri commissioner of higher education. "Everything is now geared to improving teaching and learning."

"Those dollars have made a real difference" at Southeast Missouri State University, in Cape Girardeau, according to Provost Jane Stephens, who said the campus has received $6 million in "Mission Enhancement" money over the last four years, as well as about $400,000 from the "Funding for Results" pool.

Some has been used to support "satellite centers" in parts of rural southeastern Missouri where there are no other postsecondary institutions. Other dollars have paid for an intensified advising program, which Stephens said has led to higher grade point averages, and a higher retention rate, among "undeclared majors."

Missouri Western State College, in St. Joseph, has used $2.9 million in "Mission Enhancement" funds to increase its freshman year completion rate from 41 percent to 52 percent in the last seven years.

FFR money has enabled the University of Missouri's St. Louis campus to strengthen its ties with nearby two-year community colleges, resulting in significant increases in graduation and retention rates, as well as higher grade point averages, for transfer students.

Dramatic changes have taken place at Truman State, where the successful student assessment program led to a change of mission in 1985, from a regional university serving 22 counties in an area of dwindling population to designation as a statewide public liberal arts and sciences institution. In 1996 the name was changed from Northeast Missouri State to Truman, although there are no particular ties to the family of former President Harry S. Truman.

Since the mission change, the campus has been attracting ever-brighter students, as judged by grades and ACT scores. Last fall the average GPA for entering freshmen was 3.73 and the average ACT score was 27; both were higher than the averages for first-time freshmen at the University of Missouri's flagship campus in Columbia.

The number of undergraduate programs has been cut from 140 to 43. Among the casualties were home economics, undergraduate teaching programs and most vocational and technical degrees. One hundred new faculty positions were added, reducing the student-faculty ratio from 22-to-1 to 15-to-1.

"We've added 100 faculty members without increasing enrollment (now about 6,000)," boasted President Jack Magruder. "No other institution has done that."

About half of Truman's 375 full-time faculty members have been hired since the mission change, and the quality of applicants has risen steadily. "We're getting candidates now from the Ivy League, Berkeley, the University of Chicago, places like that," said Garry Gordon, vice president for academic affairs.

But they don't always stay. There isn't much to do in Kirksville, a small town (population 17,000, counting the students) with a marginal farm economy. Kansas City is 160 miles away, St. Louis 200 miles. The opening of an eight-screen movie house in downtown Kirksville this year was treated as a major event.

"I'm looking for people who are a good fit for the university, people who have something to add and who will be happy here," Gordon said. "But we're in an isolated location and there aren't a lot of jobs for spouses. We have some difficulty attracting people, it's a significant concern."

Some students, especially those from large cities, find the combination of Kirksville's small-town atmosphere and Truman's heavy academic emphasis too much to take. "It's all school," one dropout wrote at the end of her freshman year. "There's nothing here except school!"

Nevertheless, U.S. News & World Report rated Truman as the best public regional university (one that provides mainly undergraduate education) in the Midwest last year. The framed magazine cover hangs outside the admissions office.

Truman was awarded a Phi Beta Kappa chapter this year, only the fourth Missouri institution to receive this distinction. (The others are the University of Missouri, Columbia; Washington University, in St. Louis; and St. Louis University.)

Student assessment has played an important role in Truman's rising visibility. Charles McClain, who introduced the program as campus president, later became Missouri's commissioner of higher education and was instrumental in pushing the mission change through the legislature.

"We wouldn't have gotten the mission change without the assessment program," said Candy Young, a veteran political science professor at Truman. "McClain's constant emphasis on quality and accountability is what did it. That made us pretty popular with the legislature."

Assessment is still very much part of life at Truman. Students are tested, interviewed and surveyed repeatedly. It is not their favorite activity, especially since few rewards or penalties are attached to the process.

"Today's students are relentlessly pragmatic," one faculty member said. "They want to know, 'What's in this for me?'"

But most students take the tests, submit to the interviews and fill out the questionnaires diligently, if not enthusiastically. "They all hate the assessment program but the responses we get are pretty good," said Nancy Asher, coordinator of assessment and testing.

It is, to repeat the campus mantra, "part of the culture here."

Faculty enthusiasm for assessment also varies, by individual and by academic discipline. "Every time I turn the corner, I see more assessment people," grumbled a young professor who, wishing to continue his climb up the tenure ladder, requested anonymity.

Others willingly devote large amounts of time and energy to the process. For instance, after the spring semester ended last May, three teams of 20 faculty members each spent a week reading almost 1,000 portfolios, reflecting the work students had done throughout their Truman careers. This, despite the fact that the portfolios make no difference in student grades, nor are they a graduation requirement (although they will be, starting in 2003).

Some campus critics think Truman is sometimes guilty of testing for the sake of testing and that much of the voluminous information gathered about students never is put to use.

Vice President Gordon agrees that "we need to use the data in better ways," to make changes in the curriculum and to improve campus life for the students.

But Gordon also said, "Assessment put us on the map; there's no way we would abandon it. Because of assessment, the state recognized that we could handle a statewide role as a public liberal arts college."

President Jack Magruder agreed.

"Is it too cumbersome?" he asked. "Maybe so, and we're always looking for ways to improve it, but people here feel the whole concept of assessment has brought us to the national table. It's been very good for this university."

 
Charles J. McClain started a rigorous assessment program as president of Northeast Missouri State University (now Truman State University) in the mid 1970s.  
Ninety miles south of Kirksville, on the 23,000-student University of Missouri campus in Columbia, there has been considerable resistance to the state assessment program. There are relatively few complaints about tests in subject majors, or in professional fields like nursing and teaching, but few faculty members or administrators have much confidence in the tests that are used to measure an undergraduate's knowledge in general education courses.

"In a research university like this, there's a real frustration" with using nationally normed tests to assess student performance, said Associate Provost Lori Franz. "We think we do much more than can be assessed in a multiple-choice test."

"The faculty culture is a little stronger here," said Gary Pike, assistant vice chancellor for student affairs. "If you tell them to do something, they'll resent it and refuse to do it."

Faculty members also feared assessment results would be used to take money away from the Columbia campus and give it to more cooperative institutions.

Administrators were not keen on assessment either. In a 1994 letter to the state coordinating board, Charles Kiesler, then the campus chancellor, described the approach as "basically flawed."

Taking their cue from the administration and faculty, students stayed away from the assessment tests in droves, despite lures that included free pizza, graduation gowns and even cash.

Over the years, this recalcitrance cost the Columbia campus at least $1 million in state incentive money (some say much more) and won few friends in the legislature.

Richard Wallace, who became chancellor in 1997, and Provost Brady Deaton decided to try to find a way to cooperate with the state program while not alienating the faculty. Deaton turned the problem over to Lori Franz, who was a professor of management before moving into the administration.

Reading the coordinating board regulations, Franz discovered that the campus could comply with the mandate by testing students on just one part of the CAAP general education test, instead of insisting that students devote four hours on a Saturday morning to taking all four parts.

The test is now given during a single 50-minute period, without advance notice (so students won't skip class that day) and, Franz said, "Ninety-five percent of the undergraduates now take it."

"That was a brilliant stroke," said Gil Porter, director of the campus general education program. "I wish I'd thought of it."

But Franz is still not a fan of general education assessment tests.

"To me, assessment is the outcome," she said. "Did our students go to graduate and professional schools? Did somebody light their intellectual fires? Do they get good jobs? Are they valuable citizens? Those are the things that count."

Student assessment has gained greater acceptance at Southeast Missouri State University, a 9,000-student campus serving 25 counties in the region where five states (Arkansas, Illinois, Kentucky, Missouri and Tennessee) come together along the Mississippi River.

"It really has become part of the campus culture," said Provost Jane Stephens. "I think we have real buy-in on this campus."

This was not always so. "I was a faculty member here when Governor Ashcroft first talked about assessment," Stephens said. "I thought he was the devil incarnate and so did many others."

Tony Duben, chairman of the computer science department, recalled that "it came from Jefferson City (the state capital) and was seen as one of those mandates from the topÉMost departments did not consider it a high priority."

But "there's been a gradual acceptance of the idea of assessment," said Rusty Curtis, professor of education and former chair of the campus assessment committee. "Some departments still don't see the value of it and don't take it seriously but most do, I think."

The current campus administration certainly takes it seriously. "The allocation of resources and other major decisions are being made in part on assessment outcomes," Provost Stephens said.

Southeast is not happy about the quality of the nationally normed tests that are mandated by the state coordinating board and would prefer to design its own. Campus officials are especially critical of the multiple-choice general education tests, which sometimes are several years old and do not, they say, adequately test a student's reading and critical thinking skills.

"We need to have assessments that make sense to faculty members and department chairs," said Vice Provost Dennis Holt. "Many of them think nationally normed exams are simply ineffective."

In some cases, Southeast has substituted its home-grown tests for national ones, resulting in the loss of about one-third of the state incentive money for which the campus is eligible.

Stephens, Holt and others are negotiating with coordinating board officials on a possible compromise that would allow Southeast to play a larger role in devising the tests, while retaining its eligibility for the incentive money.

Questions about the adequacy of national tests present a problem in Missouri, as they do in many other states. "We struggle with the fact that nationally normed tests are just not that good," said Kala Stroup, the state higher education commissioner. Yet, as another state official said, "there needs to be a way to keep score," to know how well Missouri is doing compared with other states.

Other tests have been developed that some experts consider to be better measurements of student reading and critical thinking. For example, in the late 1980s, Educational Testing Service developed the "College Outcomes Program" for the state of New Jersey, part of which required students to read and analyze a passage, then write several short essays about it. But the test was expensive because it could not be scanned by machine, each test had to be read individually. After three years, in the midst of a budget crunch, New Jersey dropped the program.

Another problem in Missouri and elsewhere is that some campuses collect vast amounts of student assessment data but then don't use it to improve either the curriculum or methods of instruction.

"Are we testing for accountability, so we'll look good to some governor or legislative committee, or for improvement?" asked a University of Missouri faculty member. "I can't see that much of the information we gather is being put to good use."

"It's both," Robert Stein responded. "If you're continuously improving, and you can demonstrate it, then you're also being accountable."

 
  Library building at Truman State University, a “bastion of measurement,” in Kirksville, Missouri.
However flawed the efforts to measure student learning might be, there seems little doubt that they will continue. The movement for accountability, for higher standards and high stakes testing, that has preoccupied elementary and secondary schools in recent years, almost certainly will sweep through public higher education as well. Politicians, accrediting agencies and governing boards all are demanding proof that college students are actually learning something.

"Like it or not, we are moving in that direction," said Suzanne Ortega, vice provost for advanced studies at the University of Missouri, in Columbia.

E-Mail this link to a friend.
Enter your friend's e-mail address:

PREVIOUS STORY | FRONT PAGE | NEXT STORY

Top

National Center logo
© 2000 The National Center for Public Policy and Higher Education

HOME | about us | center news | reports & papers | national crosstalk | search | links | contact

site managed by NETView Communications