In 1999, a sub-committee within the English department’s curriculum committee developed course learning outcomes for two composition courses: ENG 101 and ENG 102. They based these outcomes on nationwide standards from the Council of Writing Program Administrators (CWPA). These outcomes also correspond to KU's CORE goals on written communication.

In 2008, we began the assessment project to examine the extent to which the existing curriculum structure (e.g., assignments in 101 and 102) helped facilitate student achievement of core outcomes as well as the extent to which student work demonstrated mastery of these goals. We based our assessment procedure on an assessment of one course’s outcomes developed by the University of Kentucky. As in Kentucky's case, we focused on three aspects: 1. Examination of the alignment of assignments to course goals, 2. Examination of students' and instructors' perceptions of student performance, and 3. Examination of student performance as related to course goals. Our project differed from Kentucky’s assessment plan in that we considered two courses and multiple assignments as opposed to focusing on one course.

As a first step, we surveyed students and instructors to gauge their perceptions of student performance. Next, we developed rubrics for both 101 and 102 assignment sequences. We also collected student work to assess the extent to which the assignments aligned with the course goals and how well students were achieving the outcomes. Since this was the first time we conducted an assessment of student learning (and alignment of assignment sequences), we considered the current project as a baseline form of assessment.

This portfolio describes the English Department’s process of assessing student learning outcomes in ENG 101 and ENG 102. The project involved three parts: In Part A, students and instructors from 101 and 102 completed a survey and indicated the extent to which they agreed that students met course outcomes. Part B focused on assessing the extent to which assignments (across 101 and 102) were aligned with course outcomes (same outcomes measured in Part A). Graduate student coders used a rubric to a code the assignments and score them on three levels (central, peripheral, absent). Finally, Part C examined student performance on the major assignments from 101 and 102 courses. Graduate student coders assessed student work using rubrics, which indicated the various course outcomes and aligned with outcomes measured in Part

1. Establishment of learning goals

As mentioned in the Background section, we developed our broad goals based on nationwide standards from the CWPA. We took the CWPA outcomes, which represent what first year students should know at the end of the year, and considered which outcomes applied to which of our two courses. We also modified them to fit our program’s approach to teaching writing. This resulted in 13 ENG 1019 and 14 ENG 10210 outcomes.

2. Developing rubrics

The above-mentioned goals were helpful in understanding the course outcomes that the department (and University, given that these courses fulfill general education requirements) wanted to help develop in the first year courses. To track students’ levels of performance in these outcomes, a group of faculty drafted two rubrics for each course (ENG 101 and ENG 102). The first rubric assessed the extent to which each goal was central to the selected assignment sequences. If the goal was absent from the assignment, it received a score of 0; if the goal was peripheral to the assignment it received a score of 1; and if the goal was central to the assignment, it received a score of 2. The second rubric11 assessed student performance on the major course assignments (i.e., selected student work on major assignments). If the student work insufficiently met a learning outcome, it received a score of 0; if the student work competently met a learning outcome, it received a score of 1; and if the student work excellently met a learning outcome, it received a score of 2. Thus for each outcome, we developed three levels of performance by considering the types of performance that we were likely to see as students progressed through these courses.

3. Assessment of assignments

I collected (electronically via Blackboard) all major writing assignments from a stratified random sample of instructors teaching ENG 101 and 102 courses in Fall 2011. No identifying information was associated with any instructor’s course material, as this was an assessment of the program and not the individual instructor. I made these assignment sequences available to a summer assessment team composed of four English graduate students and the FSE (First- and Second-year English program) intern. This assessment team evaluated the 101 and 102 course assignments and assignment sequences by using the corresponding rubrics.

4. Student and instructor ratings

We created two surveys to assess students’ and instructors’ perceptions of student performance. As part of the student survey, ENG 101 and 102 students rated their perceptions of achieving each learning goal on a scale of 1 (strongly disagree) to 4 (strongly agree). In the instructor survey, instructors of the two courses indicated the extent to which they thought that students demonstrated each learning goal (on the same 1-4 scale as in the student survey).

5. Assessment of student work

I created a stratified random sample of instructors and randomly asked students from their courses to allow us to use their materials. We received IRB approval for the study and had students sign permission forms allowing us to use their writing. Instructors provided us copies of ungraded student materials. No identifying information was associated with any student’s portfolio. I made the student portfolios available to a summer assessment team, composed of five English graduate students. This assessment team evaluated the ENG 101 and 102 student portfolios by using the corresponding rubrics.

1. Assessment of assignment sequences

The first set of analyses focused on examining the extent to which assignment sequences reflected or measured learning goals.

ENG 101

Results indicate that goals 1-4 were central to all assignment sequences, followed by goal 5 (central to 90% of assignments) and goal 6 (central to approx. 80 % of assignments). Goals 8 and 11 were central to approximately 60% of assignments and goal 10 was peripheral to about 50% of assignments. To summarize, goals 1, 2, 3, 4, 5, 6, 8, 10, and 11 were central to at least 50% of the assignment sequences.

Goals 7, 9, 12, and 13 were either peripheral and/or absent from a small percentage of the assignments. No goal was absent from all assignment sequences.

ENG 102

Results indicate that goals 3, 4, 5, 6, 8, 11, 12, 13, and 14 were central to at least 80% of the assignment sequences. The rest of the learning goals were either peripheral to or central to some of the assignment sequences. One exception, goal 9, tended to be peripheral to 60% of assignments yet central to or absent in 40% of assignments (20% each).

Overall, this analysis suggests that all of the learning goals were not fully central to all the assignment sequences in ENG 101 and 102 courses. However, all the learning goals were central to at least 30% of the assignment sequences when collapsing across courses. In other words, while assignment sequences measured learning goals, each assignment sequence did not measure all learning goals at the same time. This is not problematic, as we do not expect each assignment to capture all learning goals. Our interest was more in investigating if there were some learning goals that were absent across all the assignment sequences. We were happy to see that this was not the case.

Student and instructor ratings

ENG 101

We first examined the average rating scores between instructors and students. Results indicate that apart from goal 11, student ratings are higher than instructor ratings. Thus, students believe they achieve goals to a greater extent than instructors do. In the case of goal 11, students and instructors had consistent ratings in perceptions of student performance (average score difference was less than .1). The frequency distribution reflects this consistency in rating scores for goal 11: about 90% of students and instructors agreed that students were meeting this goal. The discrepancy between student and faculty ratings was most evident for goal 7 (average score difference equaled to .69). The frequency distribution shows that 50% of instructors disagreed that students met this learning goal, in contrast to approximately 15% of students. In other words, a majority of students (90%) believed that they met this learning goal in contrast to half of the instructors.

ENG 102

As in 101, results indicate that student ratings are higher than instructor ratings. Thus, students believe they achieve goals to a greater extent than instructors do. Unlike 101, there was no goal for which the average score difference between students and instructors was less than .1. At the same time, there was also no goal for which students and instructor scores were greater than .4. In other words, while students had higher ratings than instructors, this discrepancy was not large. Thus, students and faculty had more consistent ratings in ENG 102 compared to those in ENG 101.

3. Assessment of student work

ENG 101

Results indicate that students demonstrated the highest levels of learning on goals 11 (1.35/3.00) and 13 (1.27/3.00), and lowest on goals 2 (.97/3.00), 4 (.93/3.00), 5 (.90/3.00), and 7 (.93/3.00). However, even in the case of goals 11 and 13 (which showed higher levels of learning), scores demonstrate that on average, students were unable to demonstrate excellent levels of learning (equivalent to a score of 2).

ENG 102

Results indicate that students demonstrated the highest levels of learning on goals 2 (1.13/3.00) and 8 (1.20/3.00), and lowest on goals 11 (.52/3.00) and 12 (.59/3.00). Once again, even in the case of goals 2 and 8 (which showed higher levels of learning), scores demonstrate that on average, students were unable to demonstrate excellent levels of learning (equivalent to a score of 2). Moreover, lower scores in 102 ratings were lower than those in 101: specifically, the lowest scores in 101 were closer to 1, demonstrating that on average students tended to demonstrate competent levels of learning. However, the lowest scores in 102 were much lower than 1 (approx. .5), showing there were several students who demonstrated insufficient levels of learning. Finally, these scores were inconsistent with the student and instructor ratings: for example, on average, students and instructors agreed that students met goal 11. However, examination of student work reveals that approx. 50% of the student work reflected insufficient level of learning for this particular goal.

Following the analysis of the project, we conducted workshops for instructors in the program to reflect on the results of the project.

We particularly focused on goals where there was a discrepancy between instructor and student ratings (e.g., surveys that examined people’s perceptions of the extent to which students achieved each learning goal). For instance, while student performance indicated that students achieved learning goal 13 in ENG 101 (i.e., Critique their own writing and revise to improve global qualities -focus, development, organization- as well as local qualities), students and instructors differed in their rating scores. Students perceived that they were meeting this goal to a higher extent when compared to instructors’ perceptions. In other words, instructors did not perceive students to be effectively demonstrating this goal even though student performance suggested that students were demonstrating this goal in their work. In the workshop, we asked instructors to reflect on why they had lower ratings on such goals. Responses from instructors posited that faculty may be focusing on student deficit while students focus on improvement; students see all of their drafts and know all of the changes they have made, and since the last unit focuses on revision, students were aware of what they were in the process of revising while they were completing the survey. A contributing factor could also be differences in how raters assessed improvement and how instructors grade revision projects. Raters compared a draft of the paper to the final version and rated improvement, while most instructors use a previously graded draft to compare to the revised draft. This may focus the grading on changes suggested by the instructor and minimize changes that originate with the students. Since revision is so central to the learning outcomes for this course, we will emphasize how to look for student improvement in training and workshops.

We also focused on goals for which students demonstrated low levels of performance. For instance, we considered goal 11 in ENG 102 (i.e. Learn and use at least one system of documentation responsibly). Results indicated that 50% of student work demonstrated insufficient level of learning for this particular goal. However, instructor and student ratings suggested that they agreed that students were able to meet this particular goal. In other words, there was a discrepancy between student performance and perception of student performance (by instructors and students). Discussions with faculty and graduate students suggested that this discrepancy could be due to the rubric that we used to measure student performance. The rubric assessed the extent to which students used in-text documentation correctly. However, the instructors of this course indicated that they emphasized rhetorical uses of documentation systems (e.g., when does one need a system of documentation, how does one integrate it in the text), rather than instructing students to use the same system in all their assignments. Moreover, analysis of assignment sequences suggests that assignments effectively capture this rhetorical use of documentation style. This suggested that students were actually demonstrating this goal in their work, and students and instructors also perceived that the former were meeting this goal. However, because the rubric did not effectively align with the goal, the current results indicate low scores on student performance. This is a relatively easy issue to address, as it does not have anything to do with training of students or faulty assignment sequences.

Across these workshops, we focused on ways in which we could 1. modify rubrics to capture student performance more effectively, 2. change assignment sequences that did not effectively capture learning goals, and 3. discuss learning goals with students so that they could understand what they meant and how they could achieve them.

We are now using the information from the assessment and workshops to revise GTA training and create professional development workshops addressing the issues revealed by the assessment. During the 2014-2015 year, we will also examine the assessment results to create a staff workshop that examines the ways in which the outcomes from the two courses reveal connections between the courses to refine our ideas about how the courses act as a learning progression.

By using the data gained from this assessment project, we were able to identify revisions that will result in a better learning experience for the students and to a stronger first year experience at KU.