Enhancing Feedback on Student Learning—Betsy Brand Six (2010)

Lawyering Skills (fall syllabus (pdf), spring syllabus (pdf)) is a required course for all first-year law students. As the title suggests, the course introduces law students to basic lawyering skills, including legal research, legal reasoning, and legal writing. It also includes an orientation, before law school begins, that tries to prepare students for all of law school. In orientation we cover topics such as a review of the U.S. legal system, an introduction to the common law system, and the concept of precedent. We also talk to students about how to prepare for and get the most out of their other first-year courses. In orientation, we teach students in a large lecture format. During the year, however, students are divided into smaller classes of about 20 to 25 students each.

At its most basic level, the course tries to begin the process of teaching students how the law works. The goals of Lawyering Skills I are for students to begin to understand how to find the law, how to think and reason like a lawyer, and how to write a legal memorandum. The course, including Lawyering Skills II taught in the spring, is the only required course that teaches students how to research and write legal documents. Most of the other courses in law school include at least some component of teaching students how to think and reason like a lawyer. As a result of this, much of the course focuses on teaching students how to research and write. The challenge is that to research and write well, a student needs to understand how to think and reason like a lawyer.

Another challenge is that some students come to law school with weak basic writing skills. It can be difficult to tell whether some students failed to understand the material or were just unable to articulate that understanding (or both). The concern is that students who are slower to master legal reasoning skills continue to fall further behind because of the cumulative nature of legal studies. They then may be at risk for failing the bar examination required to practice law.

Compounding this problem is that most law schools, including KU Law, have a forced mean. Thus students’ grades, at least in part, depend on how well they do in relation to their classmates. When I grade an assignment, my focus has often been on how well students do in relation to others, rather than on how well they have mastered a particular skill in the assignment. Relative grading creates several problems:

1. Students can get discouraged. Even though they may learn a great deal, their grades may stay the same if their fellow classmates also learn at the same rate.
2. I am not forced to think about whether students are learning what I am trying to teach them.
3. I am not forced to evaluate on a criterion basis what justifies a particular grade. Thus, it is not always necessary for me to determine why a student has done poorly: Did the student fail to understand the material or was he just unable to articulate his understanding in writing or both?

My goals over the year were to try to address some of these concerns. I wanted to try to find ways to better determine the specific sources of students’ struggles. I wanted to pass along that information to students, as well, and also help them see where they improved. I also wanted to evaluate how well I was teaching the specific objectives of the course.

I made two changes to my course. The first was to use clickers in the classroom. The second was to use a type of grading rubric I call a learning progress chart.

Clickers

My objectives for using clickers included:

1. evaluating whether students had mastered the material to determine whether we needed to spend more time discussing the material in class; and
2. engaging all students in the classroom learning by forcing them to participate and get real feedback on what they knew and did not know.

I used clickers in a variety of ways. The most common use was to ask students a question about the content of the reading or other assignment for class. Sometimes these were just simple questions about the subject matter of the reading (pdf). Most often, however, I used clicker questions to ask students to apply what they had learned (pdf). Then, based on how well they answered the question, I would either move on or spend time discussing the answer to the question.

I also used clicker questions to remind students of earlier material that was necessary to master before students could master new material that was the subject of class that day. On several occasions, for particularly difficult concepts, I would ask questions at a later date that would review material covered earlier in the semester. I even once used clickers to ask students the same, difficult questions that had been on an exam in the previous semester. Then we would discuss how we needed to use that information in what we were doing in class or on class assignments.

I used clickers quite frequently to ask students their opinions (pdf). In the past, I had done this with a show of hands but noticed that students were often influenced by their neighbors’ responses. Clickers allowed me to get a truer gauge of student opinion. In the spring semester, all students represent the same hypothetical client in a lawsuit. One sample opinion question I used was to ask students if they thought they were likely to win their case. Learning how fellow students felt about their case helped students evaluate their own opinions and gauge whether they might have stronger or weaker arguments than they thought they did.

More background about how I used clickers:

KU faculty members can require students to purchase and register clickers for their classes. I did not have my students purchase clickers but instead used the money I received for participating in the Best Practices Institute along with additional funding from the law school to purchase clickers. I then handed out clickers at the beginning of each class for which I planned to use them and collected them at the end of class.

I had students answer questions anonymously. I chose to do this because it made using clickers administratively simpler. I also wanted the clickers to help students learn, not cause them additional stress (law school involves plenty of stress). For many of my objectives, it was not necessary for me to know how each particular student answered the question.

Learning progress charts

My objectives for using the learning progress charts include:

1. allowing students to see their progress, or lack of progress, over the course of the semester or year;
2. allowing me to evaluate whether students are learning what I am trying to teach them; and
3. allowing me to isolate, for myself and for the student, those areas that are particularly difficult for a each student.

I attempted to create learning progress charts that I could use when grading most or all of the writing assignments in the course. I created one for the fall semester (pdf)14, when students write objective memoranda, and a similar but separate one for the spring semester (pdf)15, when students practice persuasive writing. In the past, I had used grading rubrics that were specific to the particular writing assignment and to the underlying subject of the assignment. The purpose in creating the progress charts was to allow the students and myself to see the progress students make from assignment to assignment and over the course of the semester or even the year.

I also wanted to create positive steps in the learning process. So, rather than focus on what the student failed to do, I tried, as much as possible, to focus on the positive achievements students make as they learn. So, for example, rather than saying that a student put a preview map in the wrong place, the charts acknowledge that a developing student includes a preview map. The good student also puts the preview map in the appropriate place and the very good student’s preview map is in the order the student follows in her discussion.

Students are given the chart in advance of each assignment, so they can use it to help them understand what is expected of them and what I will grade them on. Note that I do not specifically use the chart to give students a grade. There are no points on the chart. There is, of course, a general correlation between the grade and the chart. The process of filling out the chart does focus my attention on those specific things I have identified as important and I have sought to teach students in the course.

The chart itself evolved over the course of the year. For example, when I created the chart for the fall semester, I first created specific course objectives and tailored the chart to those objectives. I evaluated what I wanted to accomplish and what I thought was realistic to accomplish in a course taught in the first semester of law school. One category I did not include on the chart was whether the student’s analysis was good, logical, and sound. Yet I found this inevitably factored into their grade on an assignment. I included a category for this in the chart for the second semester.

Clickers

I found that clickers were very helpful in evaluating what students did and did not understand both before and during class. In the past, I would ask questions in class, either calling on volunteers or selecting a student to answer the question. The problem was that I often assumed that everyone or most students would have answered the same way. If the student knew the answer, often the case when I called on a volunteer, I would move on relatively quickly. If the student did not know the answer, I would explain more, even though everyone else in the class may have understood the point. Clickers allowed me to essentially call on everyone at the same time. There were many instances throughout the semester where I was surprised by the responses of students. There were instances both where more students answered correctly than I would have supposed and instances where more students answered incorrectly. I was then able to use that information to make class more efficient and focus on teaching and reinforcing difficult concepts.

Clickers also helped students understand what they did and did not know. When someone else answers a question, there is a human tendency to assume that, if asked, we would have given the correct answer. Using clickers forced students to evaluate whether they really did know the answer. Anecdotally, I had several students tell me that they thought they had mastered particular material, only to discover through the clicker questions that they had not. At the end of the semester, I asked students, with a clicker question, whether they agreed with the statement: “I like clickers because using them gave me instant feedback about whether I knew the material.” Of the 40 students in my two classes, 31 (77.5%) agreed with that statement, 8 (20%) were neutral, and only 1 (2.5%) disagreed.

Learning progress charts

Because I have never attempted to evaluate how much students have learned before, it was impossible for me to evaluate in any meaningful way how the learning progress charts affected student learning. I was able, however, to use them to evaluate how much students learned over the course of the semester and the year. The result, while not surprising, was a bit disappointing.

Most students were able to learn and improve on the same assignment. We give students several writing assignments over the course of the year. For many of these assignments, we have students rewrite their first draft. On students’ very first written assignment, there were understandably many mistakes. For example, many students had trouble organizing their writing. Unlike in many other disciplines, the law-trained reader expects the rule to come first and then expects the application of that rule to follow. On the first assignment, 39% of the students violated that rule at least once, either by putting the rule after the application or by combining the two in the same sentence. On the rewrite of that first assignment, only 18% made this same error.

Students did, however, have difficulty transferring these skills to new, more complex assignments. On the final writing assignment of the first semester, 41% made some error in the organization of the application and the rules. On the final assignment of the second semester, 38% of students made the same error. (Note that I did not have the same students in the fall and the spring semesters).

Gathering specific data on how students performed in different categories confirmed for me what I had understood only anecdotally before: Different students struggle to master different skills. So while a C student was more likely to make the mistake of putting her application before her rule, not all C students made this error and some A and B students did. Some students struggled to understand the material, some students struggled to express their understanding in writing, and some struggled with both. Using the chart helped me identify what each student’s strengths and weaknesses were and allowed me to convey that information to the student.

This information and other similar data was very useful to me in evaluating my own teaching and helping me think about whether I am achieving my course goals.

For examples of student work in this class, please check out the links below (all pdfs):

• A paper 1
• A paper 2
• B paper 1
• B paper 2
• C paper 1
• C paper 2

Clickers

Overall I was quite pleased with the benefits of using clickers. Rather than guessing or assuming what concepts students were failing to understand, I was able to simply ask everyone and then focus the class discussion on those concepts. Students felt that clickers were helpful in giving them feedback about what they did and did not understand but, because they answered anonymously, were able to learn this in a relatively low stress environment. A majority of my students thought using them was fun.

Because of the time it took to select and purchase the clickers, I did not get them until February 2010. I plan to use them more extensively in orientation and in the fall semester, when basic foundational concepts are first learned and mastered. In orientation, when we have more students than clickers, I plan to put students in groups and each group will have a clicker. I think clickers will be very useful in helping me and my students know whether they understand key concepts.

Using clickers did present some challenges. The first time I used them, I tried to ask too many questions. There is time involved in asking a question and waiting for the answer. I learned that, at least for me and my style of teaching, it really was not efficient to ask more than five questions in any one 55-minute class. It also takes time to create good questions. I did not use clickers in every class. My hope is that over time I can continue to add more questions and refine those that I have to make them better.

Learning progress charts

The learning progress charts were very useful in helping me understand what students were learning and what students were struggling with. Much of what I learned confirmed what I had thought based on experience. In particular, I confirmed that students struggle to transfer what they learn on a simple assignment to a new one, particularly when the new assignment is more complex. I plan to use this information to focus my teaching on helping students understand this difficulty and practice transferring these skills.

Over the years, I have made many changes to my teaching. I was never able to determine with much certainty whether those changes actually improved student learning. By taking the information from the learning progress charts and entering it into a spreadsheet, I am now able to more accurately compare how students collectively perform from class to class and semester to semester and better determine whether changes in my teaching affect their mastery of basic and complex skills.

The charts themselves continue to be a work in progress. As I used the charts to give feedback to students, I discovered flaws in the charts or different or better ways to express the information I was trying to convey. I will continue to revise the charts to improve the value of the feedback they provide to students.

Creating the chart forced me to think about my specific course objectives. Using the chart has also caused me to think about those objectives and about how realistic and achievable they are. I will continue to reevaluate what I teach as I reevaluate my objectives.