Building KU's Teaching and Learning Community

Comparing Student Learning in Online, Large Lecture, and Small Discussion-based Courses—Paul Atchley

Overview

A professor uses a capstone assignment to compare student learning across three offerings of a research methods course: 1) online, 2) face-to-face large lecture, and 3) face-to-face small discussion-based delivery.

Background

In this portfolio, I examine the extent to which student learning is impacted when a course is administered in an online setting as compared to a traditional face-to-face lecture format and a small discussion format. While online learning can be an effective way to teach, one barrier to implementing online courses is developing ways to assess their effectiveness relative to face-to-face equivalents using common measures of learning outcomes. To address this concern, I examined student performance on a capstone exercise for three different versions of a research methods in Psychology course: 1) online, 2) face-to-face large lecture, and 3) a face-to-face small discussion-based delivery. The small discussion class served as the baseline and as an intervention for course improvement on a single goal where students were not doing well (understanding the limits of scientific inquiry).

Implementation

I designed the online and the face-to-face lecture courses to be as similar as possible. Both had the same organization of material, used the same book and lecture materials, and had students engaging in the same exercises and peer grading activities. The only difference between these two offerings was that in the lecture based offering, I had face-to-face interactions with the students. In the online course, students still interacted with one another but I did not have to facilitate these interactions.

To assess student learning across three offerings, I designed a capstone assignment in which students completed a written application for the Director of Clinical Trials position at a pharmaceutical company. The assessment consisted of six open-ended essay prompts that were designed to encourage students to integrate their classroom discussions and research experiences. Finally, to examine how a smaller discussion-based offering of the course could provide a base line to understand student learning, a colleague administered the capstone assignment to a small discussion-based class at the beginning and end of the semester.

Student Performance

First, it appears that students in the discussion class (since we used pre-test and post-test only for this offering) did improve across the semester: scores on the pre-test and post-test improved across the semester. Second, the average scores across the face-to-face and online courses indicate that there were no obvious decrements in students’ learning for the online course when compared to face-to-face lecture course. The only difference in performance was on Question 6 of the assignment, where students were asked to evaluate the strength of claims made in an online advertisement for a competing anti-depression product. Students in the online version of the course scored higher on this question than students in the lecture class. The higher score for the students in the online course may have been an artifact of selection bias, such that students who were more comfortable learning using online technology may have enrolled in the summer online course, relative to students in the more traditional lecture course. One common aspect across the online and lecture based offerings was the low performance on the question designed to test whether students understood the limits of science. In the small discussion-based version of the course, student performance showed an encouraging trend toward increased upper-level proficiency on describing the limits of scientific inquiry.

Reflections

I feel I can safely say that going online does not have to result in a decline in student learning. The key seems to be the interactivity of the assignments. While there was no online baseline for a course that is non-interactive, or a pure lecture-only course (the current course was about half lecture and half interactive assignments), my sense is that the interactive exercises are what lead to student learning. When students were asked to indicate the experiences that led to their learning, some talked about working in labs or other activities, but most of them outlined their own in-class work as evidence of their knowledge and experience. In the future, additional work will concentrate upon improving these experiences and designing even more effective interactive tasks.


^Back to top^

Background

Although there has been an institutional call to develop more online courses, the call for online delivery has been met with healthy skepticism. In addition to a concern about the time required to implement and manage online offerings, many faculty express concern about the quality of online courses relative to more traditional course delivery methods.

While online learning can be an effective way to teach, one barrier to implementing online courses is developing ways to assess their effectiveness relative to face-to-face equivalents using common measures of learning outcomes. The reliance on stand-alone assessments for different course types can lead to poor estimates of learning outcomes across different delivery methods. To address this concern, I examined student performance on a capstone exercise for three different versions of a research methods in Psychology course: 1) online, 2) face-to-face large lecture, and 3) a face-to-face small discussion-based delivery that served as the baseline and intervention for course improvement on a single goal where students were not doing well (understanding the limits of scientific inquiry). This work was done in collaboration with Sarah Bunnell, formerly of KU and now at Ohio Wesleyan.

In this portfolio, I will focus on three major themes. First, I examine whether student learning, as measured by a capstone assessment, is impacted when the course is administered in an online setting as compared to a traditional face-to-face lecture format. Second, I describe how I used a rubric to analyze the common capstone assignment. And third, I describe how student performance in the online and face-to-face lecture courses was used to modify a discussion-based offering of a course and support the development of enhanced understanding of research methods.

There were 170 students in the large lecture course, 17 students in the small online course, and 38 students in the small discussion-based course. A majority of the students were sophomores and juniors, and most of the students had enrolled in this course to fulfill a Psychology major requirement.

Table of Course Make-up

The course goals (similar across all three settings) are listed below:

  • Understand ethical issues in science
  • Determine the validity of scientific and non-scientific claims and formulate additional explanations
  • Identify the limitations of science and other forms of inquiry
  • Identify problems with scientific studies
  • Properly formulate a question to which the scientific method can be applied
  • Design a scientific study to answer a properly formulated question
  • Interpret the results of a study to produce valid claims of fact

^Back to top^

Implementation

The online and the face-to-face lecture courses were designed to be as similar as possible. Both had the same organization of material, used the same book and lecture materials, and had students engaging in the same exercises and peer grading activities. However, in the online version, no live or recorded lectures were provided, and I kept my interactions with class members to a minimum, in an attempt to avoid biasing the final assessment by spending more time per student in the online class. Peer grading was done asynchronously in the online version instead of face-to-face, though it was guided by the same rubrics and reflection questions as the in-class version. Exams were online rather than in the classroom (though mastery quizzes were delivered online in both environments). The only difference between these two offerings of the course was that in the lecture-based offering, I had face-to-face interactions with the students. In the online course, students still interacted with one another but I did not have to facilitate these interactions. I used the peer assessment option in Blackboard and students had two days to upload assignments. They then received the grading rubric, reflection instructions, and assignments from two other students, all of which they turned in after two more days.

As stated earlier, to assess student learning across the three offerings, I designed a capstone assignment (pdf) in which students completed a written application for the Director of Clinical Trials position at a pharmaceutical company. The assessment consisted of six open-ended essay prompts that were designed to encourage students to integrate their classroom discussions and research experiences. For example, some of the questions on this capstone assignment asked students to design an ethical study of a new drug, evaluate the evidence for a drug’s effectiveness, and discuss the types of questions that an empirical psychological study can and cannot address.

The assignment was aligned to the course goals with specific focus on ethics, research evaluation, and designing studies, as well as assessing any experience the students had from the course or other sources that were relevant to research methods. Sarah Bunnell, my OWU colleague, and I designed a rubric to evaluate student learning on this assignment.

Finally, to examine how a smaller discussion-based offering of the course could provide a baseline to understand student learning, my colleague administered the capstone assignment to a small discussion-based class at the beginning and end of the semester. Further, this course was re-designed to emphasize understanding the limitations of science. Sarah taught the small course (38 students), in which she was able to interact more closely with each student.


^Back to top^

Student Performance

To examine the impact of online-based offerings of the course on student learning, Sarah and I first compared student performance across the large face-to-face lecture based offering and the online offering of the course. A factor analysis suggested that questions 2 and 4 be collapsed into one factor (called “Learning”) and questions 1, 3, 5, and 6 form another factor (called “Knowledge”).

Learning

Knowledge

First, it appears that students in the discussion class (since we used pre-test and post-test only for this offering) did improve across the semester: scores on the pre-test and post-test improved across the semester.

The average scores across the two offerings (face-to-face lecture and online) on these two factors indicate no significant differences in student performance, suggesting that there were no obvious decrements in learning for the online course when compared to the face-to-face lecture course. The only difference in performance was on Question 6 of the assignment, where students were asked to evaluate the strength of claims made in an online advertisement for a competing anti-depression product. Students in the online version of the course scored higher on this question than did students in the lecture class. The higher score for the students in the online course may have been an artifact of selection bias, such that students who were more comfortable learning using online technology may have enrolled in the summer online course, relative to students in the more traditional lecture course. Regardless, this result points to the power of meeting students where they are in terms of positively impacting the type of performance that is generated in an academic setting.

However, performance on Question 5 (designed to test whether students understood the limits of science) was lower than I had hoped, given that one goal of the course was to teach that concept. Further, it was not clear without a baseline if students were improving as a result of the course or simply performing at an intermediate level due to experience in other psychology courses. The latter question was of great concern given the preponderance of juniors and seniors in the large lecture course. To address this concern, my colleague redesigned the discussion-based offering of the course to emphasize the limitations of science.

To examine the effectiveness of this discussion based offering on understanding the limits of scientific inquiry, we looked at scores on Question 5 for the small discussion-based course. Our analysis revealed no statistically significant differences, perhaps due to the relatively small sample size of the discussion-based class (see Table 2, below).

But the distribution of scores shown in the figure below shows an encouraging trend toward increased upper-level proficiency on describing the limits of scientific inquiry in the small discussion-based version of the course, though this should be examined more closely in future work with more data from smaller sections of the course.

Examples of student work, with comments on Question 5

Example of expert work (pdf)  
Comments

Example of intermediate level work (pdf)
Comments

Example of novice level work (pdf)
Comments


^Back to top^

Reflections

Data show that students did fairly well on most questions from the assessment. In the small discussion-based course, student performance improved on the measure over time; their baseline performance was below the end-of-semester performance in the lecture and online courses. While overall performance was not at the highest level for the majority of students, performance was at the intermediate level generally, which meets expectations for an introductory research methodology course. It should be noted that the course does not have a concurrent lab, so these measures reflect learning through in-class exercises and lectures. Used in conjunction with laboratory activities, I would expect students to improve further.

What about teaching online versus in the classroom? It is important to emphasize that the in-class course is a large, lecture-based course, so student/instructor interaction is not as common as it would be in a small, discussion-based course. So, the course might more closely match the typical online experience where a student is more disconnected from the instructor. I would like to emphasize that the online course used the same materials as the lecture course, and the same assignments, with peer interactions and peer grading done online. Because of these aspects of the online and lecture courses, the two versions of the course are probably about as comparable as one can get.

Given those caveats, I feel I can safely say that going online does not have to result in a decline in student learning. The key seems to be the interactivity of the assignments. While there was no online baseline for a course that is non-interactive, or a pure lecture-only course (the current course was about half lecture and half interactive assignments), my sense is that the interactive exercises are what lead to student learning. Students were asked to indicate the experiences that led to their learning. While some talked about working in labs or other activities, most of them outlined their own in-class work as evidence of their knowledge and experience. In the future, additional work will concentrate upon improving these experiences and designing even more effective interactive tasks.

Contact CTE with comments on this portfolio: cte@ku.edu.


^Back to top^
Author: 
Discipline: 

Portfolio & Poster Search




List of all portfolios & posters

One of 34 U.S. public institutions in the prestigious Association of American Universities
44 nationally ranked graduate programs.
—U.S. News & World Report
Top 50 nationwide for size of library collection.
—ALA
23rd nationwide for service to veterans —"Best for Vets," Military Times
KU Today