Report on Assessment of Critical Inquiry Revealed by Senior Papers

 

In our institutional assessment plan, critical inquiry is one of our major goals for the overall learning of our students.

 

Seven faculty participated in the assessment working group who assessed student written work for the Critical Inquiry goal:  Tim Burns (Government), Paty Rubio (Foreign Languages), Dawn Riley (Education), Rik Scarce (Sociology), Tim Harper (Management and Business), Laura Paul (UWW), and Stephen Murray (Religion).  The group met on two occasions, first, to develop a common understanding of their purpose and, on a subsequent date, to draw conclusions based upon their reading of a variety of student papers.  In between, they read and evaluated the papers.

 

This assessment had the following purposes:

 

1.      to determine whether our students’ work addressed the critical inquiry outcomes in a manner appropriate for Skidmore graduates and to identify outstanding examples of student work that did address any of the learning outcomes within the goal;

 

2.      to determine whether the critical inquiry goal as drafted could be assessed by examining student papers, and, if not, to recommend revisions to the goal; and

 

3.      to make recommendations regarding the process of assessing the critical inquiry goal.

 

 

I.                   The Critical Inquiry Goal

 

 

Critical inquiry, as defined within our Assessment Plan, involves using various modes of inquiry and interdisciplinary perspectives or methodologies to conceptualize, investigate, and derive meaning.  It implies that learners are active learners, self-motivated learners, and learners who understand the ambiguities and uncertainties of achieving absolute knowledge, as well as the implications of various courses of action.

 

Skidmore students who demonstrate evidence of their being critically-inquiring humans:

 

 

 

 

 

 

Discussion:

 

“Critical inquiry” is not defined within any of the curriculum documents of Skidmore College, including the Catalog, yet the abilities specified by this goal are outcomes that faculty often refer to in their discussions of the purposes of a liberal education at Skidmore.  We often talk about wanting our students to draw upon interdisciplinary knowledge or perspectives in addressing various problems and issues, to recognize the weaknesses in their arguments or recognize what they do not know, and to employ the tools of various disciplines either in their continuing learning or in their future actions.  The critical inquiry goal is our attempt to make these desirable outcomes of a liberal education overt.  Below we discuss how well we think the goal as drafted does define those outcomes.

 

“Critical inquiry” is not synonymous with “critical thinking,” although the two overlap in a variety of ways.  While many standard definitions of critical thinking exist in the literature, no nationally accepted standard definition of critical thinking exists since each discipline emphasizes different aspects of thinking.  Critical thinking may involve a wide variety of skills, such as evaluating, comparing and contrasting, inferring, applying knowledge, or drawing conclusions.  In many ways, critical thinking resembles the concepts of motherhood or patriotism.  Each of us knows what we mean by it, but few of us agree completely upon any common definition.

 

The critical inquiry goal is intended to describe a set of characteristics that capture the intellectual capabilities of what we expect from Skidmore graduates.  As such, it encompasses much of what we intend students to learn in our general education program, but it also assumes that the general education program is only a beginning and that students’ learning is cumulative, growing through learning in the majors, through elective coursework, through participating in co-curricular activities, and in experiences outside of the physical confines of the College.  The goal is also intended to be applicable to a wide variety of student work, from traditional research papers to art works.  We hope that our initial effort to define what we intend by “critical inquiry” will begin a conversation at Skidmore that will lead to more overt understandings by both faculty and students of the overarching purposes of a liberal education at Skidmore.

 

 

 

 

 

 

 

II.                Observations Based upon the Review of Student Work

 

We read 100 papers:  9 Art papers, 41 Social Science papers, 6 Natural Science papers, 32 Humanities papers, 4 papers from the pre-professional areas, and 8 papers we could not place in any particular disciplinary area.  The papers included traditional research papers, science reports, and reflective essays.  We read five senior theses as part of this assessment since we were interested in the thinking revealed by the papers more than the quality of the writing per se.  Each reader read between 14 and 17 papers, depending upon whether they read senior theses, and submitted an evaluation report on each.

 

Constraints:

 

Despite attempts to collect work from all graduating seniors, we were only able to collect a limited number of papers from the students themselves or from a small number of faculty who graciously copied work that their seniors had submitted in their courses.  As a result, we read all submissions, either as samples to develop common understandings of our purposes or for the assessment itself.

 

From the beginning, we recognized that no single paper would be likely to encompass all of the learning outcomes specified by the goal, but we wanted to understand whether and how well the papers reflected any of those outcomes.  We did not have the assignments that the students were given, and so we are unaware of the expectations that the individual faculty had in making their assignments or the purposes of the students in writing their papers.  We could assume much about the assignments, but we could not be certain of our assumptions.  We did not know the extent to which early drafts of the papers were reviewed by the faculty or by other students.  Some of the papers were senior theses, and, as such, were undoubtedly subject to intensive review prior to the final draft.  They were included in our sample because we were interested in the thinking revealed by the papers, more than the writing ability, which was addressed overtly by the writing assessment group.  Finally, despite our attempts to collect samples of papers from all of our seniors, we did not accomplish that, and so our conclusions may not reflect the full range of senior written work.

 

 

Summary Observations:

 

  1. The theses of most papers were weak and a few non-existent. When present, many theses were either too all-encompassing or self-evident and thus did not lead to critical discussions or reflect a depth of understanding or interpretation.  Even when good theses were present, student writers often lost track of those theses and drifted in their development of an argument.  Occasionally a student discovered an insightful thesis by the end of a paper, but had not concentrated upon that thesis during the paper and had not revised the paper to reflect the insights that the thesis might have encompassed.  The lack of well-developed theses in so many papers indicates to us a corresponding lack of clear focus or purpose.  Evidently, many students do not approach their papers with a sufficiently critical intent.  That is, too many papers read like reports rather than well-considered explorations of a topic.

 

  1. Most students do not recognize opposing or alternative arguments or points of view.  If students have considered alternative perspectives, those apparently have been set aside when writing the paper.  In an attempt to develop an unambiguous argument, most simply do not recognize the breadth of perspectives reflected in the scholarly debates and inquiries of our disciplines.  Readers commented on such matters as unexamined assumptions or uncritical acceptance of the jargon of the discipline.  While few students critiqued their own conclusions, this limitation may be an artifact of the types of assignments they were given.

 

  1. The papers overwhelmingly reflect disciplinary perspectives and knowledge and seldom integrate knowledge or perspectives from other disciplines.  While this may be the natural and desirable result of study within any one discipline, it does not reflect the ostensible intent of Skidmore to promote interdisciplinary study.

 

  1. The ability to use sources meaningfully varied greatly across the papers.  This may reflect the nature of the varied assignments that the students were given.

 

  1. While it was not the purpose of this assessment to evaluate the writing ability of our seniors, readers commented that weak arguments often seemed to coincide with such aspects of writing as weak organization or weak syntax.

 

  1. The variety of topics, issues, and subject matter reflects the diversity and breadth of courses at Skidmore and, as such, speaks well to the intellectual opportunities available to our students.

 

  1. The assignments have led students to employ a wide range of tones and styles, ranging from pieces reflecting the styles of the popular media to those of scholarly journals.

 

  1. Some papers were truly outstanding in the writers’ depth of thinking and ability to engage readers, but the majority of papers were, as summarized above, limited in a variety of ways.

 

  1. Despite the critical comments above, we recognize that our students are still developing as thinkers and writers.  We cannot expect all of our students to be fully mature thinkers and polished writers.  The more challenging an inquiry task may be, the more likely the student is to demonstrate weaknesses in writing.

 

III.             Is the Critical Inquiry Goal (Including Its Outcomes) Appropriate for Skidmore?

 

In general, the readers found the critical inquiry goal and its outcomes to appropriately reflect the desirable overall learning of students who graduate from Skidmore, but the readers questioned the exact wording of each of the outcomes. 

 

For the purposes of this assessment, the readers chose not to determine whether a particular paper employed both quantitative and qualitative methodologies, but chose to determine whether students used methodologies appropriate to the subject.  They recognized that, because they were not experts in all the disciplines represented at Skidmore, they could not be certain that their judgments about the methodologies were accurate.  Nevertheless, they agreed that graduates of Skidmore should be able to use both quantitative and qualitative methodologies.

 

The readers revised outcome two as follows (italics indicate additions):  “Interpret, evaluate, integrate, and use information discerningly from a variety of perspectives or sources.” 

 

They struggled to find a way to understand the outcome “transform the results of the inquiry into judgments and actions,” and so they revised it as follows:  “transform the results of the inquiry into refined judgments.”  The readers chose to drop “actions” from their review since they felt that they had no basis upon which to judge actions that may have resulted from the individual studies of the students.  Please see the discussion in the next section on the use of rubrics to understand the readers’ response to their revisions.

 

The readers added one more outcome:  “Demonstrate openness to reason or intellectual honesty/probity.”  After reading all the papers and upon subsequent reflection, they felt that this outcome might be contained within the outcome “interpret, evaluate, integrate and use information discerningly . . . .”  or the outcome “. . . take into account alternative points of view.”  Once again, please note the discussion that follows on the use of rubrics.

 

 

IV.              What Process Should Be Employed to Assess the Critical Inquiry Goal?

 

 

Given the constraints of this assessment process, we recommend the following:

 

1.      Samples of student work should be drawn from the total body of senior work.  Very few students submitted their work voluntarily.  Subsequent appeals to faculty to submit the major papers of their seniors led to uneven submissions, with some faculty submitting all their seniors’ work and some selecting only the best work, which defeats the purpose of an honest assessment.

 

2.      To assess all the learning outcomes specified within the critical inquiry goal, a portfolio system would best enable this to occur.  We recommend that the College continue to explore the implementation of an electronic portfolio system and requirement that students compile portfolios of their work.  The obligation to compile a portfolio and submit samples of work for assessment purposes should be placed upon the students.

 

3.      If reflective essays are to be assessed as part of the critical inquiry goal, then students should be guided in how to write a reflective essay so that it meets the criteria of the goal or reflective essays should be evaluated using those appropriate criteria that faculty determine.  We have a wonderful opportunity to begin implementing such guidelines for reflective essays in the new Scribner Seminars since students will be asked to begin drafting reflections upon their academic goals.

 

4.      The critical inquiry goal and its outcomes should be discussed and debated among all faculty.  The result may be a total revision of the goal and its outcomes, but that result can then be communicated to students in a variety of forums and media so that they understand what an education at Skidmore can mean.

 

5.      The assessment team of faculty should involve faculty from each of our disciplines so that particular disciplinary constraints or intentions can be communicated to all team members.

 

6.      Members of the assessment team should read papers in common, perhaps with at least two of the team members reading the same papers and comparing their evaluations.  In our assessment, other than when we established our initial understanding of the outcomes, we each read different papers and so did not debate our judgments about individual papers.

 

7.      The questions about what each of the learning outcomes means can best be addressed by developing rubrics.   As defined in our assessment handbook on our assessment website, a rubric is a method of classifying and categorizing student behaviors or products along a continuum.  The rubric defines the characteristics of an outcome or goal from the mosst desired characteristics through acceptable characteristics to unacceptable characteristics.    Should we want a numerical compilation of our assessment results, rubrics contain within them simple scales.  The faculty conducting an assessment can develop their rubrics based upon the sample of student work available to them, not simply by adopting a rubric that someone else has created for a different set of student work.  [Rubrics also can serve another very important function, that of providing a guide as students attempt to fulfill an assignment.]

 

8.      We wondered whether, aside from senior theses and honors projects, students produced their best work during the second semester of their senior year.  Some believe that many students begin to relax academically by the time of their second semester.  Should we only collect work from the Fall Semester, or should we allow and require students to submit work from any of their years here as long as the students believe the work addresses the outcomes?

 

 

V.                 Conclusions:  Questions for Continuing Consideration

 

We submit the following questions for the assessment task force to consider and recommend that they be considered and discussed by all faculty:

 

1.      Does the critical inquiry goal reflect what we expect of graduates who have had a liberal education at Skidmore?  Can we find a way to discuss this goal and refine it so that it does reflect our intentions?

 

2.      How can we collect a broader representation of student work to assess?

 

3.      Should we ask that assignments be submitted along with the products of those assignments?

 

4.      Can we implement a portfolio system to which students must submit work?  Should this be an electronic portfolio system?  How can we best implement such a system?  If we employ a portfolio system, do we need to see the assignments that have led to the student work?

 

5.      How can we best assess student work in the arts if the critical inquiry goal is to encompass such work?

 

6.      Since the best student work that we reviewed tended to be senior theses or honors projects, should all students be required to complete a major project of comparable effort and depth?

 

7.      Should we overtly teach the skills that would lead to the critical inquiry learning outcomes that we desire, and, if so, in what courses and at what level or levels?  If we believe in these outcomes, how can we best teach toward them?  If, for example, we believe that students should be self-reflective or self-critical, how do we best guide them to become so?

 

8.      What are the implications of the critical inquiry assessment results for our professional development efforts?  Should we offer more workshops on specific pedagogical strategies?