Handbook: Indirect assessment methods

Definition:

Indirect assessment methods require that faculty infer actual student abilities, knowledge, and values rather than observe direct evidence. Among indirect methods are surveys, exit interviews, focus groups, and the use of external reviewers.

• Surveys: Surveys usually are given to large numbers of possible respondents, usually in writing, and often at a distance.
• Exit interviews and focus groups: Exit interviews and focus groups allow faculty to ask specific questions face-to-face with students.
• External reviewers: External reviewers are usually representatives of the discipline and usually are guided by discipline-based standards.

Advantages:

• Indirect methods are easy to administer;
• Indirect methods may be designed to facilitate statistical analyses;
• Indirect methods may provide clues about what could be assessed directly;
• Indirect methods can flesh out areas that direct assessments cannot capture;
• Indirect methods are particularly useful for ascertaining values and beliefs;
• Surveys can be given to many respondents at a time;
• Surveys are useful for gathering information from alumni, employers, and graduate program representatives;
• Exit interviews and focus groups allow faculty to question students face to face;
• External reviewers can bring a degree of objectivity to the assessment;
• External reviewers can be guided either by questions that the Department wants answered or by discipline-based national standards.

Disadvantages:

• Indirect methods provide only impressions and opinions, not hard evidence;
• Impressions and opinions may change over time and with additional experience;
• Respondents may tell you what they think you want to hear;
• The number of surveys returned are usually low, with 33 percent considered a good number;
• You cannot assume those who do not respond would have responded in the same way as those who did respond;
• Exit interviews take time to carry out;
• Focus groups usually involve a limited number of respondents;
• Unless the faculty agree upon the questions that are asked in exit interviews and focus groups, there may not be consistency in the responses.

Variations:

• Electronic surveys: Surveys can be sent out as attachments to email messages. Another method involves having a survey appear on a student’s screen when the student first logs on. Some programs have made these surveys short, asking only one or two questions at a time so that the student is more likely to respond seriously.
• “Literary or Art Criticism” model: An external reviewer might actually write a review of the materials that he or she reviews, applying his or her own standards or those developed by external groups. This method is likely to be more subjective and may not answer the questions that the faculty want answered—unless they ask the reviewer to address them. On the other hand, the informed subjective opinions of a national expert in the field may provide valuable insights and advice regarding the academic program.
• Institutional research data:
• Percentage of students who go to graduate school;
• Statistics on job placement;
• Retention;
• Courses selected by students;
• Faculty/student ratios;
• Percentage of students who enroll in study abroad;
• Enrollment trends;
• Diversity of students in the program.

NOTE: These types of data provide various forms of evidence about your program, but do not provide actual data about student learning. They may, however, give you various data on other aspects of program success.

Guidelines:

Surveys:

· To encourage responses, keep surveys short;

· Ask only for information that you want to use;

· Ask for more than Likert scale and attitudinal responses:

o Simulations: “What if . . . ?” “Imagine that . . . .”

o Open-ended: “Describe the hardest problem that you had to address in our program.”
”If you had time to re-do one of your research papers, which would it be, and what would you do differently?”

“If you could design a new course for our program, what would it be and how would it work?”

· Do not use a lot of surveys with the same students;

· If you want to correlate responses with certain characteristics of the students, code surveys so that you can disaggregate specific groups even while you keep the individual’s responses confidential;

· Gather responses in a timely manner.

Focus groups:

· Be alert to the power of the interviewer – a Department faculty member might intimidate the students;

· If possible, use an interviewer from outside of the Department;

· Have only a few key questions – develop follow-up questions as the interview proceeds;

· Be alert to the student who dominates the conversation – ask others for their opinions;

· Target your focus group population, e.g., seniors, students who have recently finished the introductory course, students who chose the thesis option;

· Consider, when possible and appropriate, focus groups with other populations, e.g., employers, parents, undecided freshmen;

· Ask open-ended questions;

· Ask questions that require specific examples rather than just attitudes;

· Keep the focus group small – 5 to 10 individuals;

· Let the group know how long the focus group will last before they attend it;

· Record conversations for later transcription or use a note-taker in addition to the focus group leader.

A sample questioning pattern for a focus group session:

1. (If you don’t know each other, let’s start by introducing ourselves. I’m . . . .)
2. What was your overall impression of (this program)?
3. What was the most difficult assignment or learning exercise in this program?
4. What assignment or learning exercise was most useful in helping you to learn what the program required?
5. Has you been able to learn anything that you’ve learned in this program outside of the program itself? How?
6. If you could give any advice to the faculty teaching in this program, what would it be?
7. Is there anything I should have discussed with you that I omitted?