Portfolio Assessment: An Alternative to Traditional Performance Evaluation Methods in the Area Studies Programs

The primary mission of the REES (Russian, East European, and Eurasian Studies) MA is to provide a comprehensive interdisciplinary introduction to Russia, East European and Eurasia with language competency appropriate for a wide range of professional careers in the government, private-sector, NGOs, business, international law, law, or media. It is also designed to meet the needs of those students preparing for advanced graduate study in a particular discipline.

Upon completion of their two-year MA degree program at CREES (Center for Russian, East European, and Eurasian Studies), students are expected to:

• demonstrate broad knowledge of the history, cultures, societies, and politics of the region;
• acquire language proficiency at the appropriate level of professional competence;
• apply a variety of disciplinary and interdisciplinary approaches to the study of the region; and,
• attain an on-site experience and analytical skill appropriate for an area expert.

To assess the extent to which students met the above-mentioned goals, the REES MA program used a traditional three-hour closed book exam administered at the end of the students’ final semester. Students were required to answer at least three questions from different disciplinary divisions within the REES MA (political science, history, literature and arts, geography/economics/business, and philosophy and religion). Those exams were then graded by a committee of three graduate faculty affiliated with CREES.

In Spring 2010, the CREES Executive Committee members agreed that the traditional written MA examination failed to adequately demonstrate the extent and depth of student knowledge. Although there is an MA reading list, this list has little relation to the widely varying course work that REES MA students complete. Students are often at a loss as to how to prepare adequately for the exam. Moreover, because the exam itself does not align with the abovementioned program goals, it is difficult to assess student learning and examine the extent to which students meet program goals.

Finally, this traditional exam form of assessment did not capture the change in learning across the two-to-three years of the program, as it is a form of summative evaluation. Summative assessments are often relatively easy to integrate into program assessment; however, they often fail to adequately show the depth and extent of students’ integrative learning and cultural competency, which are hallmarks of the interdisciplinary nature of the area studies programs. By shortchanging the process of learning in favor of the products of learning, they also leave little room for students’ input into the learning process or self-reflection on their learning.

To address the issue of alignment between the program's goals and its assessment-instruments, and to incorporate a form of evaluation that is more process based (compared to being more summative), the program restructured the MA requirements and replaced the exam component with the student portfolio. The present portfolio documents the impact of this comprehensive student portfolio on learning and the authenticity of the program’s assessment and compares it to past exam performance.

At CREES, a framework for assessing the REES MA program consists of the following parts:

1. An MA portfolio provides a direct measure of student learning in the graduate program with a particular focus on students’ integrative and interdisciplinary learning, acquisition of language skills, and cultural competency;
2. A graduate exit questionnaire measures students’ perceptions of their own learning in the graduate program, their assessment of the program, and resources offered to them during their graduate experience; and,
3. Other separate assessment instruments, such as pre- and post-surveys of academic events, teaching and professionalization workshop, and reflection letters that the recipients of scholarships are required to submit at the end of the term supported by the award are used as additional measures of the quality of resources offered in the program.

I will specifically focus on the MA portfolio and describe the various components and assessment methods.

A portfolio is a, “systematic collection of student work and related material that depicts a student's activities, accomplishments, and achievements in one or more school subjects. The collection should include evidence of student reflection and self-evaluation, guidelines for selecting the portfolio contents, and criteria for judging the quality of the work [emphasis added]. The goal is to help students assemble portfolios that illustrate their talents, represent their writing capabilities, and tell their stories of school achievement...” (Venn 2000, 530-531). Portfolios can consist of a wide variety of materials: teacher notes, teacher-completed checklists, student self- reflections, reading logs, sample journal pages, written summaries, audiotapes of retellings or oral readings, videotapes of group projects, and so forth (Valencia 1990).

Two types of portfolios are widely used in teaching practices: process portfolios and product portfolios. A process portfolio documents the stages of learning and provides a progressive record of student growth. A product portfolio demonstrates mastery of a learning task or a set of learning objectives and contains only the best work. The CREES MA portfolio combined both elements (process and product) and therefore served as a method to assess student learning over time as well as mastery of program goals. The portfolio not only contained a capstone paper (that demonstrates mastery of several learning tasks) but also included examples of written work that students developed at the start of their MA program.

CREES electronic portfolio

The six elements of the CREES MA portfolio are:

• three samples of scholarly writing,
• a capstone paper,
• a synthetic essay, and
• a professional essay.

These are designed to cumulatively represent student academic progress related to the program’s goals and provide authentic evidence of student self-evaluation, reflection, and goal-setting.

Samples of scholarly writing: demonstrating student progress and accomplishments. These samples include at least one original research paper and can also include a series of three policy briefs, a historiography paper or other synthetic essay with a critical literature review, an annotated literary translation, or other lengthy, critical work approved by the student’s committee. All must have been completed in courses taken in three of the five required REES disciplines.

Capstone paper: demonstrating the mastery of the subject matter, language skills, and interdisciplinary and integrative learning. Students work on their capstone papers in the last semester of the MA program. It is an example of interdisciplinary work that also showcases the use of the student’s chosen target language. The length of the capstone paper is an approximately 7500-word text, plus bibliography, where at least 25% of the work must be in the student’s target language.

Synthetic essay: demonstrating student integrative learning, language skills, self-assessment of progress in learning, and self-evaluation of learning. The synthetic essay is an intellectual response to the student’s work through the penultimate semester of area studies coursework, as well incorporating outside reading from the MA Reading List in the chosen region of specialization. It is the student’s opportunity to reflect upon what he or she has learned, while drawing conclusions about how various disciplinary ways of thinking intersect and inform each other.

The synthetic essay addresses the following themes or questions:

1. Based on coursework at KU (including language courses) and the MA Reading List for the student’s chosen region of specialization, what are the major regional themes or problems? An answer to this question need not give equal emphasis to each of the five major disciplinary perspectives, but it should not be limited to, say, the student’s major subject discipline. The student seeks to define topics common to several disciplinary approaches or overlapping themes. To illustrate major points, the student uses specific examples from primary sources, whether from coursework or the MA Reading List.
2. How has the study of various disciplinary approaches affected the way the student views this special area? How, for example, does normative or statistical analysis inform humanistic study, and vice versa? The student uses specific examples, both from coursework and the MA Reading List.
3. Finally, what are the student’s conclusions concerning the area studies degree? What are the two or three main concepts or ways of thinking that the student has absorbed? How does the student make sense of the REES interdisciplinary experience?

Professional essay: demonstrating student self-assessment of learning, mastery of skills, and preparedness for professional and academic careers. This essay is an extended first draft of a future job application letter in the student’s field. If the synthetic essay looks back, the professional essay looks forward, picking up where the synthetic essay left off and addressing the following questions:

1. What is the student’s career objective?
2. How has the REES MA prepared the student to move into a related career (please address, among other things, the REES curriculum, faculty, programming, resources)? If the plans include continuation of graduate study, the student will want to discuss how the area studies degree has offered good preparation and helped to shape the student’s interests for further study at the PhD level.
3. How will the area studies approach impact the student’s future life and work?

In addition, REES MA portfolio documentation contains explicit guidelines for selecting the portfolio content and is accompanied by rubrics (pdf) for assessing all elements of the portfolio. These guidelines and rubrics were available to faculty and students. After a student submits all materials (electronically), a committee of three graduate faculty members evaluates the elements of the portfolio using various rubrics.

In addition to the written work, the portfolio also includes a 90-minute oral examination at the end of the student’s final semester. Approximately 30% of questions in this exam address the three research papers (or equivalent projects); 40% deal with the MA capstone seminar paper; and 30% engage with the student’s synthetic and professional essays. Students are allowed to take the exam a second time in the event they fail the exam the first time. The committee uses a rubric to evaluate student performance in the oral examination.

References:

• Venn, J. J. 2000. Assessing Students with Special Needs (2nd ed.). Upper Saddle River, NJ: Merrill.
• Valencia, Sheila. 1990. "A Portfolio Approach to Classroom Assessment: The Whys, Whats, and Hows." The Reading Teacher, 43 (4): 338-40.

Assessment Process

To assess the effectiveness of portfolios on student learning, three independent raters examined the components of the MA portfolios of the first cohort of MA students from Spring 2012. We specifically used the capstone paper as a comparison with the traditional exam (since both were summative forms of assessment).

We used two AAC&U rubrics to measure student learning: Integrative Learning (pdf) and Intercultural Awareness (pdf). Integrative Learning is an understanding and a disposition that a student builds across the curriculum and co-curriculum, from making simple connections among ideas and experiences to synthesizing and transferring learning to new, complex situations within and beyond the campus. Intercultural Knowledge and Competence is "a set of cognitive, affective, and behavioral skills and characteristics that support effective and appropriate interaction in a variety of cultural contexts” (Bennett, J. M. 2008).

Both rubrics aligned with REES program goals and therefore helped evaluate the extent to which students met these goals. We compared scores on the rubrics’ dimensions to ratings of the traditional exams (using the same rubrics) of the MA students who graduated in Spring 2011. The two samples included four electronic portfolios from Spring 2012 and five final exams from Spring 2011.

To compare the two formats (capstone paper vs. final exam) on student learning, we first examined the mean scores of each element in the Intercultural Awareness Rubric. 

Results show that the capstone paper demonstrated significantly higher levels of learning on five of the six elements. Specifically, students demonstrated higher levels of:

1. Knowledge and engagement with multiple cultural frameworks;
2. interpreting cultural experiences from more than one worldview;
3. articulating their understanding;
4. asking complex questions about other cultures; and,
5. demonstrating interest in knowing more about people from different contexts and cultures.

Moreover, this implies that after changing to the portfolio format, students were more likely to meet the REES goal on demonstrating broad knowledge of the history, cultures, societies, and politics of the region.

Next, we compared the average scores in each element of the Integrative Learning rubric. In this case, results show that the capstone papers demonstrated higher levels of student learning in two of five elements: Connection to Discipline and Transfer of Skills, Abilities, Theories, and Methodologies. This resonates with the REES goal of “applying the variety of disciplinary and interdisciplinary approaches to the study of the region.” In other words, results showed that the capstone paper was more effective in capturing students’ abilities to demonstrate their interdisciplinary approach and make connections across multiple perspectives.

The frequency distributions (see above) show that more students demonstrated capstone levels of learning in their final papers when compared to the final exams.

In addition to comparing the capstone papers with the final exam, we also assessed their performance on their synthetic essays.

Results show that students were able to successfully reflect on their learning and draw conclusions about the ways in which various interdisciplinary ways of thinking intersect and inform each other.

Results show that the capstone papers are a more effective method of assessment, when compared to the final exams. More students demonstrated higher levels of learning in the final capstone papers compared to the traditional final exams. Moreover, given that the capstone papers are the end products of the portfolio, results suggest that the portfolio overall is a better tool to develop as well as capture student learning.

Not only is the portfolio better aligned with program goals, but it also allows for capturing student learning across the years in their MA program. The portfolio process itself develops certain skills and raises student awareness about the merits of inter-disciplinary education, enhances their cultural competency, and provides an opportunity for self-assessment of their accomplishments. Based on the above results, the REES MA program will continue to use the portfolio method.

Finally, results show that these samples of student work demonstrate their cultural competency/integration skills and, when these data are collected over time, can be used to measure learning in MA program during specific periods.

Recent course changes

In the first round of portfolio development, we ran into the problem with students not posting some elements of the portfolio online (in Blackboard) in a timely manner. In a few instances, the essays were uploaded on Blackboard a night before the oral defense or not uploaded at all. To address this issue, we have changed the language of the portfolio guidelines to make the placement of all elements of the portfolio on Blackboard on the due dates mandatory.

We also added a “reading journal” to the portfolio for assessing students’ language proficiency. The student portfolio relies on student self-assessment and self-reporting of their academic progress. However, for the purpose of their language assessment, we relied on the language instructors’ letters. To make this part of assessment consistent with the “spirit” of the portfolio, we approved a reading journal. In this journal, students document their ongoing development of reading skills, both on topics of general interest and topics related to their CREES coursework and, in due course, capstone paper. The final entries in a student’s Reading Journal should show the highest level of reading skill they have achieved in the target language. The completed reading journal is then added to the MA student’s portfolio.