Using the Capstone Course for Assessment of Learning in the Sociology Major

Catherine White Berheide
Skidmore College

As assessment initiatives become more widespread, higher education is increasingly moving beyond externally developed standardized tests to assess student learning in majors. Senior capstone projects are often proposed as qualitative instruments for assessing student learning because their greater flexibility adapts better to the unique circumstances of each department and because they reveal actual student performance. Capstone projects, therefore, can be a vehicle for assessing how well students have met the goals of the sociology major. For example, is the student able to formulate empirical research questions? To describe important theories in a substantive area? To summarize current research in that area? To generalize appropriately? This paper looks at how capstone courses are being used to assess student learning in the sociology major.

To be useful in assessing the major curriculum, capstone projects must be amenable to comprehensive group level assessment. Disciplines, such as sociology, need models for how to use senior research projects to evaluate the department's success in fulfilling program goals. The fundamental question is what are the criteria upon which these senior projects should be evaluated? How are the goals of a sociology major translated into a set of criteria that can be used to evaluate student projects to assess the program's success in meeting its learning goals for the major? This paper addresses the question of how to assess the effectiveness of academic majors, particularly the sociology major.

Assessment in Higher Education

Currently within higher education, the mandate to engage in assessment, particularly for accreditation, refers to assessing student learning outcomes. Assessment, however, is not limited to the individual student level; it can also occur at the classroom and the program level. Faculty resist the mandate to engage in program assessment for a variety of reasons. Faculty may be suspicious of the way administrators and other authorities might use assessment. Pratto (1996:122) concludes that "assessment for purposes of improvement that is summative and carried out with the direct involvement of the individual being assessed is the least susceptible to abuse." Faculty may also be reluctant to add an additional time-consuming obligation to their workload, especially one for which they are not rewarded. Perhaps more importantly, though, faculty may resist program assessment initiatives because they are skeptical about the assessment techniques being used (Hartmann 1992).

In response to the demand for accountability, particularly from state legislatures, higher education has tended to rely on quantitative assessment tools that allow national norming, such as standardized tests. Cross and Steadman (1996) as well as Palomba and Banta (1999) reflect the positions of many faculty when they criticize these tests for failing to measure the full range of learning that goes on in college. The second of the American Association of Higher Education's (2001) "9 Principles of Good Practice for Assessing Student Education" states that, "assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time." To gain a more comprehensive picture of the outcomes of higher education, the AAHE as well as Palomba and Banta call for using a diversity of methods, so that schools do not simply assess those aspects of a college education that are easy to measure quantitatively.

This critique of quantitative techniques does not mean, however, that they have no place in a department's assessment plan. Quantitative research methods have significant advantages. They can be high on reliability and validity. They can allow for national norming and benchmark comparisons. They can allow schools, programs, or students to be compared. In addition to standardized tests either nationally or locally designed, commonly used quantitative techniques include surveys of seniors and alumni as well as analysis of measures of student success, such as grade point averages, graduation rates, percent going on to graduate school, etc.

Institutions often adopt standardized tests to measure student learning outcomes to take advantage of these strengths of quantitative methods as well as the fact that they do not require as much faculty time as other assessment tools, especially if they use a pre-existing national test, such as an ETS Major Fields Test. Such tests allow students to demonstrate what they know in their major field in comparison to a national standard. The problem is that the major at a particular school may not be designed to cover the same areas as the Major Fields Test, for example, covers. Thus the test may not accurately measure what students have learned. Even worse, the national test may lead some schools to "teach to the test." Others may decide to develop their own test to measure their departmental learning goals for majors more accurately. This option, though, reduces the likelihood of national comparability and increases the time faculty have to spend on assessment initiatives.

To gain a more comprehensive picture of student learning, some programs have turned to qualitative techniques that can provide a richer set of data than quantitative ones. As is often true of qualitative research methods, they may have less reliability than quantitative methods, such as standardized tests, but they often have greater validity. Qualitative techniques, like quantitative ones, require that the academic program define its goals and objectives and then identify appropriate methods for measuring whether those goals and objectives have been achieved. Qualitative techniques commonly used in higher education for program assessment include exit interviews, focus groups, reflective essays, synthesizing projects, comprehensive exams, and student portfolios.

As long ago as 1992, Hartmann proposed that sociology departments use the bachelor's paper, that is a synthesizing project, to assess the program as a whole.

In short, he argues that the performance of sociology programs ought to be measured by their student's ability to do sociology and that that is best done through a capstone paper.

The Capstone Course in Sociology

The Association of American Colleges report, The Challenge of Connected Learning, and those in the companion volume, Reports from the Fields, note that a key element in study in depth is some type of integrative capstone experience. The second volume (1991b), Reports from the Fields, presents the reports of the thirteen disciplinary task forces on study in depth in undergraduate majors. The fourth recommendation in the expanded version of the sociology report, Liberal Learning and the Sociology Major (Eberts, Howery, Berheide, Crittenden, Davis, Gamson, and Wagenaar 1991) is that sociology majors should have at least four levels in a sequence of courses rather than a "ferris wheel" where an introductory sociology course is the most students need for access to the rest of the sociology curriculum. The four levels the sociology taskforce (Eberts et al. 1991:15-16) recommended are

In the overview seminar, students engage in a systematic review of the discipline, consisting of various reading and writing assignments to integrate, critique, and apply sociology, with an emphasis on methods and theory. This type of course could help students prepare for a comprehensive examination whether oral or written, whether a nationally-normed standardized test or a locally designed one.

(See Eberts et al. 1991 and Wagenaar 1997 for further discussion of capstone courses in sociology.)

While the capstone can take a variety of forms, graduating seniors benefit from building a learning community as they discuss the common issues arising from independent research projects or internships. Therefore the capstone should involve a senior seminar in addition to whatever independent work might be required of students. Such a capstone is a critical ending point for a major carefully constructed not only to expose students to the discrete aspects of sociology as a discipline, but also to give them the opportunity to demonstrate their in-depth knowledge of the field.

Some Advantages of Using a Capstone Paper for Program Assessment

The synthesizing paper that students write in a research-based capstone course requires them to define a research problem, create and implement a research design, and analyze and interpret data. The advantage of such papers for assessment is that they take students through the process of doing research. The student gains additional knowledge and skill from the assessment experience while the department gains information it needs to improve its major.

Hartmann (1992:126) argues that the "additional learning experience for the student" is one of the eight advantages of using the bachelor's paper as an assessment method. I adapted the following additional advantages for using a capstone project for assessing the sociology program from Hartmann's list.

The latter, of course, is also one of the main weaknesses of using a capstone project for assessment. Without such scores they are unlikely to satisfy the external stakeholders who want quantitative nationally normed data with which to hold higher education accountable.

Using content and performance standards, however, faculty can rate capstone papers using quantitative scales. According to Heywood (1989), using more than a single evaluator and a single scale can increase the reliability of senior projects as assessment tools. (See Heywood for examples of scales that can be used to assess papers and projects in a variety of disciplines.) Hartmann (1992:127) argues that, "the key is that efforts to improve the reliability of assessment should be based in what is first of all a valid assessment project, such as the bachelor's project." He goes on to warn that, "the urge to quantify assessment beyond what can reasonably be required for reliability should be resisted."

Six Steps in Using a Capstone Project for Assessing the Major

The first step in using a capstone project for program assessment is to identify the program's goals and objectives (the third principle of good practice for assessing student learning, according to AAHE). Assessment involves measuring performance against goals. The literature on higher education contains lengthy discussions of goals (e.g., Bowen 1997, Ehrlich 2000, and Young 1997). Wagenaar (1991:93), for example, specifically develops ten goals for the undergraduate sociology major that he presents as a starting point for departments to use in developing their own. He rejects the position that "sociology is too fragmented a discipline, and there are too many sociologies to list a comprehensive set of goals." In the same volume of Teaching Sociology, Hazzard (1991) offers a set of learning goals for introductory, intermediate, and advanced courses. Departments need learning goals for students majoring in their discipline. Rather than stated as generalities, each goal a department adopts needs to specify the "such that" statements that would indicate specifically how achievement of each objective can be assessed. For example, if a department adopts a goal on sociological research methods, exactly what must the student demonstrate to show compliance with the objective?

Once a department has agreed upon a set of learning goals, it must then use them to inform the content of the curriculum and the major. Does the curriculum specifically reflect these objectives in how it is organized and taught? How should the department organize the major(s) and teach each course so that students achieve its learning goals? Are students made aware of the basic goals and how each course connects to them? Once the department has answered these questions collectively, faculty should include learning objectives for their courses on their syllabi, showing students how these courses contribute to the department's overall goals. Several departments have created matrices of department goals by courses that indicated which courses work on achieving which goals. (See Appendix A for an example of a Goals by Courses Matrix.)

Next, these goals need to be translated into content standards and performance standards: that is the skills and knowledge the student should achieve and the level at which the student should demonstrate them (Glatthorn et al. 1998). For example, one of my department's content standards is that, "The sociology major should be able to describe major theories in selected substantive areas of sociology." California State University at Sacramento (see Dean Dorn's piece, An Electronic Assessment Portfolio at California State University – Sacramento, in this volume) uses a very simple set of performance categories for assessing papers in the portfolios of sociology majors:

These content and performance standards set the benchmarks for the program as well as the individual student. These standards should be developed into rubrics for measuring how well students demonstrate that they have met the goals of the major. (See the webpage for the College of DuPage Outcomes Assessment Committee for a rubric developed by the sociology faculty at the College of DuPage.)

Once standards are developed, the department selects individuals to review the capstone projects. These multiple evaluators need to look for patterns of strengths and weaknesses across all the papers (or a sample of papers). They need to report the curricular implications of these patterns to the sociology program faculty. As Hartmann (1992:126) notes, "if a cohort of students is not meeting expectations with regard to theory, research skills, writing, policy relevance, or whatever else the department has agreed is important, curricular reform is in order."

The final step, then, is to use the data collected about the performance of the program to provide feedback to improve the major. For example, after my own department surveyed seniors to ask them to assess their sociology major as part of the AAC's study in depth project (1991b), the students' comments floored the faculty. The seniors said that the department ought to require statistics for the sociology major. As a result, we instituted a statistics requirement in addition to the methods requirement that already existed. Assessment, therefore, is not an end in and of itself, but rather a means to an end. The end is the improvement of learning at the individual, program, and institutional levels.

How Programs have Used Capstone Papers to Assess Sociology Majors

This approach to program assessment is labor-intensive for faculty as well as students. Some schools are committed to the capstone project to assess student performance but do not take the next step to use it for program assessment. The papers themselves, like the student work in a senior art show, can be displayed to external audiences with a flourish so as to say, "Ta da! Look at what our students can do." These displays can be very impressive indeed, and while they may be "an unwieldy basis for external assessment, they provide the most direct and most unfiltered picture of students' capabilities" and, by extension, of program effectiveness (Hartmann 1992:128).

Some schools, including my own, go a step further and submit some of the papers for presentation at regional sociology meetings and for undergraduate paper contests. Success in getting senior projects accepted for presentation at professional meetings or in winning undergraduate prizes provides external validation of the quality of student performance and by extension of program effectiveness. Others use external evaluators to "grade" the papers and the programs. This approach is the "thumbs up, thumbs down" approach to evaluating the capstone projects akin to a juried art show.

Best practice, though, involves going a step further, if only a baby step, to analyze the papers systematically for the evidence they provide about program quality and then to use that evidence to make curricular improvements. One step, albeit an informal and somewhat nonsystematic one, involves the "water-cooler conversation," the casual comments that the faculty teaching the capstone course make to their colleagues about what the students seem well prepared or ill prepared to do. For example, I have complained for years about the students' inability to define a sociological research question, let alone a testable hypothesis. That feedback has led the instructors in lower-level courses, including introductory sociology, to add assignments that give students multiple experiences in defining a sociological question and identifying hypotheses.

Other departments, such as Valdosta State University, are taking more formal and systematic steps. The sociology department adopts a fairly holistic approach to assessing a sample of the capstone papers every other year. According to the chair of the department, a subset of faculty reading the papers consider the conceptual depth and clarity, use of resources, and the structure of the paper.

James Madison University goes a step further. In the capstone course at James Madison, "the students integrate previous experience in the major by 'doing sociology' in the writing and presentation of a substantial paper displaying the student's sociological skills – theoretical, methodological and practical." Multiple readers use indicators derived from the department's goals to look for patterns and trends in student performance on a sample of papers from various years and then use that information to revise the sociology curriculum. The readers rate each paper from on a five-point scale from fair to excellent on each indicator that is relevant for the paper. For example, one indicator is "interprets and uses social science data" (James Madison University Sociology Department). My own department has identified the program goals that performance on the capstone project can measure, concluding that six of our ten goals for sociology majors can be assessed through their senior seminar research projects (see Appendix A). Often departments choose to tackle assessing only one or two of these goals at most per year.

The most elaborate approach, exemplified by California State University at Sacramento (again, see Dean Dorn's piece, An Electronic Assessment Portfolio at California State University – Sacramento, in this volume), involves taking the rubric the department has developed for assessing how well the department's goals for majors have been achieved and applying it to student papers. The department creates a collective department portfolio of student work by having faculty select a sample of student papers written for required courses as "examples of poor, average, and good to excellent work." The department's assessment committee then evaluates these papers to measure the performance of the sociology major as a whole. The department began by choosing to assess the success of the student papers in meeting only two of the program's eleven goals. Based on the evaluation of student papers, the department developed a list of ways to improve students' writing and their ability to apply sociological concepts (the two goals being evaluated). The department plans to analyze other goals in future years as well as to track trends in the strengths and weaknesses of student papers.

Conclusion

More and more, sociology departments are developing capstone courses. Some of them are beginning to use these capstones for program assessment. As of yet, though, only a handful have begun to evaluate systematically the students' capstone products to see what they can tell the department about the quality of its program and to help identify what needs to be changed. Using capstone projects offers significant advantages as well as some significant disadvantages compared to using standardized tests to assess the quality of sociology programs.

Sociology departments should see program assessment as a long-term process to be approached one step at a time. Each step taken, no matter how small, should lead to improvement. For example, adopting a set of goals for the sociology program requires faculty conversations that should improve the coherence of the major. In addition, developing a capstone course for the major if the department does not already have one will strengthen the major immensely. Sociology departments need to engage in systematic assessment of their majors to improve the quality of our students' performance. The major papers students write in a capstone course provide a data-rich vehicle for doing program assessment.

References

American Association for Higher Education. 2001. "9 Principles of Good Practice for Assessing Student Learning." http://www.aahe.org, June 19, 2001.

Association of American Colleges. 1991a. The Challenge of Connecting Learning: Liberal Learning and the Arts and Sciences Major, Volume One. Washington, DC: Association of American Colleges.

_______ . 1991b. Reports from the Fields: Liberal Learning and the Arts and Sciences Major, Volume Two. Washington, DC: Association of American Colleges.

Bowen, Howard R. 1997. Investment in Learning: The Individual and Social Value of American Higher Education. Baltimore, MD: The Johns Hopkins University Press.

Cross, K. Patricia. and Mimi. H. Steadman. 1996. Classroom Research: Implementing the Scholarship of Teaching. San Francisco, CA: Jossey-Bass.

Eberts, Paul, Carla Howery, Catherine White Berheide, Kathleen Crittenden, Robert Davis, Zelda Gamson, and , Theodore Wagenaar. 1991. Liberal Learning and the Sociology Major. Washington, DC: American Sociological Association.

Ehrlich, Thomas, ed. 2000. Civic Responsibility and Higher Education. Phoenix, AZ: Oryx Press.

Glatthorn, Allan A. with Don Bragaw, Karen Dawkins, and John Parker. 1998. Performance Assessment and Standards-Based Curricula. Larchmont, NY: Eye on Education.

Hartmann, David J. 1992. "Program Assessment in Sociology: The Case for the Bachelor's Paper." Teaching Sociology 20:125-128.

Hazzard, John. 1991. "Student Competencies and the Goals of the Undergraduate Curriculum: A Response to Theodore Wagenaar." Teaching Sociology 19:532-532.

Heywood, John. 1989. Assessment in Higher Education. 2nd ed. New York: Wiley.
James Madison University Sociology Department. nd. Sociology Program Goals/Indicators and Assessment Rate Form.

Kain, Edward L. 1999. "Building the Sociological Imagination Through a Cumulative Curriculum: Professional Socialization in Sociology." Teaching Sociology 27:1-16.

Palomba, Catherine A. and Trudy W. Banta. 1999. Assessment Essentials: Planning, Implementing and Improving Assessment in Higher Education. San Francisco, CA: Jossey Bass.

Pratto, David J. 1996. "Assessment Abuse by Design." Teaching Sociology 24:119-122.

Wagenaar, Theodore. 1991. "Goals for the Discipline?" Teaching Sociology 19:92-92.

Wagenaar, Theodore, ed. 1997. Capstone Course in Sociology. Washington, DC: American Sociological Association.

Young, Robert B. 1997. No Neutral Ground: Standing by the Values We Prize in Higher Education. San Francisco, CA: Jossey Bass.

Sociology Department Web Sites

California State University, Sacramento, Department of Sociology.

College of DuPage Sociology Outcome Assessment: Displaying Sociological Imagination.

James Madison University Department of Sociology and Anthropology.

Skidmore College. Courses and Goals of the Sociology Program.

Valdosta State University Department of Sociology, Anthropology and Criminal Justice. http://www.valdosta.edu/soc/