The Impact of Course Transformation on Student Learning and Success in a Gateway Electrical Engineering Course — David Johnson & Molly McVey (2019)

EECS 140, Introduction to Digital Logic Design, (or EECS 141, the Honors variant of the course) is a required, introductory course for incoming freshmen in the Electrical Engineering, Computer Engineering, Computer Science, and Interdisciplinary Computing majors, which typically enrolls 230 students per year, split between two course sections. EECS 140/141 is offered every semester as a four-credit hour course and is broken down between three hours of lecture and two hours of a required lab each week. The core curriculum includes number representation, digital coding, Boolean algebra, combinatorial logic design, sequential logic design, and working with programmable logic devices, all critical skills for success in EECS majors. The overarching goal of this course is to prepare students for subsequent courses in the EECS department (Figure 1). (Interdisciplinary Computing majors follow the same curriculum as Computer Science majors and are thus not differentiated in Figure 1). Students must complete EECS 140/141 with a grade of at least C to continue on to advanced courses in the major.

Figure 1. Curriculum map for Digital Logic Design, which prepares EECS students for degree programs, including Electrical Engineering (EE), Computer Engineering (Coe), and Computer Science (CS).

The number of students earning grades of D or F or withdrawing from the course (hereafter referred to as “DFW rate”) has traditionally been high in EECS 140/141, averaging around 25% over the history of the course, which was a major motivating factor for transforming course design. Another major catalyst for my EECS 140/141 redesign came from my experience teaching a computer science course as part of two, one-week high school summer camps. To try to keep students engaged during this summer camp, I decided to differentiate my instruction from typical lectures by providing a more active and immersive learning experience, where students learned to program a computer game. I minimized lecture time, instead allowing students to use class time for hands-on computer programming. Over the course of the camp, I realized that this change produced an unexpected and measurable improvement in overall engagement and level of performance compared to the lecture classes that I had taught.

Ultimately, this summer experience inspired me to try out some active learning practices in my teaching of EECS 140/141 during Fall 2017. During that time, I met Molly McVey, a postdoctoral teaching fellow for the School of Engineering, who encouraged me to implement a fully-flipped classroom design in the next semester (Spring 2018). Together, we applied for a C21 Course Transformation Grant from the Center for Teaching Excellence, which enabled us to hire two undergraduates to develop and create course materials, primarily complex in-class problems. Dr. McVey has also been working to collect and analyze data on student performance, DFW rates, and other results of the EECS 140/141 flipped classroom intervention. Since then, we have completed a second semester of the fully-flipped class (Fall 2018).

Through this portfolio, I hope to:

1. Make EECS 140/141 course information available for other instructors who may be interested in transforming a similar course,
2. Document the differences in student performance and retention noted between implementing active learning into the classroom (Fall 2017) versus running a completely flipped classroom (Spring and Fall 2018), and
3. Reflect on how my role as an EECS 140/141 professor has changed throughout the course transition from traditional lecturing to a hybrid lecture and active learning course to a fully-flipped classroom design.

Before the course transformation, EECS 140/141 had been taught as a traditional lecture class built on 28 major objectives. During Fall 2017, I began to transform EECS 140/141 by teaching a hybrid lecture and active learning style class. The next semester, in Spring 2018, Dr. McVey and I implemented a totally flipped classroom design, where nearly 100% of class time was spent working on problems, with all content delivery happening outside of class. A second iteration of this fully-flipped classroom model was implemented in Fall 2018.

In Fall 2017, I spent approximately 30 minutes of each 75-minute class period lecturing. After each major point, I stopped to ask for student questions. For the second half of the class, students applied lecture material to an in-class problem, which they were to complete and turn in by the end of class. A typical in-class problem was: Design a circuit that multiplies an 8-bit unsigned number by 9, using only 1 full adder. Students were encouraged to work with their peers to solve the problem, but each student was required to turn in their own work at the end of class. These in-class problems comprised 10% of the semester grade (Table 1).

Students were also assigned eight to ten additional problems to complete as homework over the week; together, these problems constituted 20% of the students’ total grade (Table 1). A typical homework problem might be: (a) Show that the circuit in Figure P3.2 is functionally equivalent to the circuit in Figure P3.1. (b) How many transistors are needed to build this CMOS circuit? During that semester, 45% of the semester grade was represented by three equally-weighted open book/note/internet, individual exams (Table 1). A typical exam question might be: Create a truth table for the logic function: f = x(x·y)(y+z).

The next semester (Spring 2018), I implemented a fully-flipped classroom design with the help of Dr. McVey. The class took place in a classroom specifically designed for active learning (five-person tables) as opposed to the auditorium-style classroom where class had been held during the previous semester. However, for the second semester of the EECS 140/141 flipped classroom, class was held in the traditional auditorium-style classroom (Figure 2a-b). To flip the course, I subdivided the 28 (30-minute) lectures into 84 modules (10 minutes each).

Differing class schedules for two sections of EECS 140/141 caused an immediate challenge in converting to a flipped classroom. One section met three times per week for 50 minutes, while the other section met for 75 minutes twice per week. To overcome this challenge, I assigned two modules per class for the 50-minute classes and three modules per class for the 75-minute classes. Each of the 84 modules featured one complex problem for students to solve in each class. In-class module problems were developed with the assistance of undergraduate research assistants who had successfully completed EECS 140/141 and had served as undergraduate teaching fellows (UGTFs) for the Fall 2017 class; these past students were well-equipped to judge the subject material and level of difficulty well for each module and did a very impressive job designing the problems.

Table 1. Breakdown of course assessments and associated percentage of each in student final grades in Fall 2017 (hybrid lecture-active learning model) versus Spring 2018 and Fall 2018 (fully-flipped model).

A typical day in the flipped classroom required everyone to be engaged. EECS 140/141 students were expected to come to class prepared for the in-class problems, after watching a 10-minute video lecture for each module (which I created with the help of the KU Center for Online and Distance Learning), reviewing provided annotated PowerPoint slides, and when necessary reviewing the course textbook. Students were also required to complete an online quiz on Blackboard prior to class to assess their comprehension of pre-class lecture materials. Quiz questions were designed to address the content of each module and represented 25% of students’ total grade (Table 1). Students were allowed to retake this pre-class quiz an unlimited number of times until the deadline before class. Students were not assigned any additional work to complete outside of class.

Figure 2a-b. EECS 140/141 students, facilitated by course instructors and UGTFs, work to solve in-class problems in a flipped classroom in Spring 2018 (a) and Fall 2018 (b).

In the classroom, UGTFs and I worked as facilitators for the in-class problem modules, spending the full class period walking around the classroom and checking in with students as they worked on in-class problems (Figure 2a-b). Facilitation tasks might consist of answering student questions, clarifying concepts, and addressing misconceptions. Students were allowed to tackle problems independently, or in groups (which they tended to prefer), but each student was required to turn in a problem set individually. Click here to see an example of an in-class problem. To ensure that each student was learning course material rather than just copying down the work of a peer, grading of in-class problems was weighted most heavily on students’ ability to demonstrate that they used the correct methods to arrive at their solution, with less weight allocated to the final answer itself (see In-Class Problems Grading Rubric). In-class problems represented 25% of the total semester grade (Table 1). Course GTAs were responsible for grading the in-class problems. In addition, students took four open-book, individual-based exams over the course of the semester, which constituted 25% of their final grade in total (Table 1). The remaining 25% of the grade was based on their performance in the corresponding laboratory section for EECS 140/141 (Table 1).

To evaluate EECS 140/141 student performance between the hybrid lecture and active learning class (Fall 2017) and the fully-flipped class (Spring 2018, Fall 2018), we compared Classroom Observation Protocol of Undergraduate STEM (COPUS; Smith, Jones et al. 2013) behavioral observations, exam performance, student feedback from mid-semester surveys, and numbers of students receiving a grade of D or F, or withdrawing from the course (DFW rates).

We used COPUS protocols to quantify how the instructor and students spent their time in class in Fall 2017 versus Spring 2018 and Fall 2018. A trained COPUS observer attended each of my EECS 140/141 sections three times over the course of two weeks to record which of 12 instructor and 13 student behaviors occurred in each two-minute interval of each class period. COPUS results did indicate that the Spring 2018 classroom was fully-flipped, with students actively working for 97% of class time and listening to the instructor for just 3% of the class time, usually for brief, pre-class announcements. Fall 2018 was similar to the first semester of the flipped classroom model, with students spending 96% of the class working, 3% of the class receiving information, and 1% of the class talking to the class (Figure 3). These results are much different from the Fall 2017 hybrid active learning and lecture class, where students spent 45% of their time working and 45% listening, on average. In addition, instructor classroom behaviors transitioned from 62% guiding and 38% presenting in Fall 2017 to over 99% guiding and less than 1% of the time presenting in Spring 2018 and 98% guiding, 1% presenting, and 1% administrative tasks in Fall 2018 (Figure 3). From these COPUS data, we see that the flipped class involved no lecture; rather, students were prepared to complete in-class work based on the pre-class materials that they had reviewed.

Figure 3. COPUS observation results for students (blue) and instructor (orange) in Fall 2017 (hybrid active learning and lecture course), Spring 2018 (fully-flipped course), and Fall 2018 (a second semester of the fully-flipped course). For students, dark blue=Talking to Class, medium blue=Working, and light blue=Receiving Information. For instructors, dark orange=Guiding and light orange=Presenting Information, and grey=Admin.

To assess student achievement of learning objectives, performance on exam questions was compared between Fall 2017 (active learning class) and Spring 2018/Fall 2018 (fully flipped class). There was a statistically significant improvement in scores on 40% of learning objectives in the fully flipped classes, on average, when compared with Fall 2017 active learning class (Mann-Whitney U). These improvements were especially important for Exams 1 and 3, where we saw significantly improved performance on nine out of 14 and nine out of 13 learning objectives, respectively (Figure 4). For Exam 2, we saw a statistically significant improvement on three out of 13 learning objectives, but also a statistically significant decrease in score for two out of those 13 learning objectives (Figure 4). On Exam 4, we saw a significant improvement in two of 18 objectives, but a significant decline in four of those 18. A major consideration on Exam 4 is that it is given as the final exam, and students already know exactly how they have to score to get a certain grade in the class, so effort may not have been consistent with the other three exams. Another potential factor that may have contributed to improved student exam performance in the fully flipped class was that students were given an average of eight minutes per question for Spring and Fall 2018 exams, but only about five and a half minutes per question in Fall 2017. This change was due to a difference in number of semester exams, with three exams given in Fall 2017 and four exams given in Spring 2018 and Fall 2018. Prior to each new semester, I reviewed videos, PowerPoint slides, Blackboard quizzes, and in-class problems associated with each exam question in an attempt to improve the score results.

Figure 4. Student exam performance on three exams, over the course of the hybrid lecture/active learning semester (Fall 2017; light green) and the fully-flipped semester (combined Spring 2018 and Fall 2018; dark green). Numbers on the x-axis represent specific learning objectives. Asterisks indicate significant differences in average score in the form of either increases or decreases. Figures are organized using the four-exam schedules used in Spring 2018 and Fall 2018; because students in Fall 2017 took only three exams, their scores are mapped by learning objective to correspond with the four-exam schedule.

During the Spring 2018 iteration of EECS 140/141, we administered a mid-semester survey to solicit student feedback pertaining to the fully-flipped classroom model. We based the survey on Likert-scale questions about the effectiveness of each course component. We also wanted to see how students were preparing for class, to understand the relative effectiveness of the video lectures, PowerPoint slides, and textbook chapters. Survey data indicated that over 90% of students felt that they came to class prepared, with 43% of students reporting that they prepared for class by reviewing the PowerPoint slides and 34% preferring the video lectures (Figure 5). Most students (90%) thought that the in-class problems were effective at helping them learn course material. Student comments further supported this result. For example, one student reported that “[they] enjoy[ed] the flipped classroom learning style, as learning the material beforehand and using the class time to make sure [they] truly underst[ood] the material [was] very effective for [them].” In another comment, a student described the benefits of learning from a larger teaching team, which included an instructor, UGTFs, and their peers: “There are so many people available to help us understand the material. One person in particular almost always explains things to me in ways that make sense and are easy to understand…personally I am happy to learn the material on my own and only use class time to clarify the concepts. It works well for how I learn.” From classroom observations, it seemed that students who experienced difficulty with one in-class problem tended not to be the same students who had trouble on another problem set. In this case, where students tend to bring different skill sets to the problem sets that they approach, we suggest that peer-to-peer instruction may help students to perform better in class than they may have when working individually.

Figure 5. Self-reported student preparation methods for class periods and quizzes during the Spring 2018 semester. Data was collected during a mid-semester evaluation for EECS 140.

The fields of engineering and computer science are known for having particularly low representation of women and under-represented minorities (Borrego et al. 2005, May and Chubin 2003). Studies suggest that active learning (particularly in the form of cooperative activities) can improve student retention in STEM classes (Felder et al. 2000, Freeman et al. 2014). Here, we measured student retention using DFW rates, and compared these rates between Fall 2017 (hybrid active learning and lecture class) and Spring 2018/Fall 2018 (fully-flipped classes). Results suggest that converting to the flipped classroom model improved retention of non-white-male students (Figure 6). In particular, we saw lower DFW rates in first-time freshmen (male and female), under-represented minority groups (especially significant among the first-time freshmen of these groups), and in overall performance (on average) among all demographic groups. By holding the instructor constant for these comparisons, we can assume that these results are a reflection of the course transformation.

Figure 6. DFW rates (%) compared among traditional lecture (Fall 2013-Spring 2017; gray), hybrid active learning-lecture course (Fall 2017; light green) and the fully-flipped course (combined Spring 2018 and Fall 2018; dark green). Results are broken into different student demographic groups. Error bars are missing from Fall 2017, because we report only one semester of data.

Future plans will include comparing DFW rates in the hybrid (Fall 2017) and fully-flipped (Spring 2018, Fall 2018) courses to a similar course Discrete Structures (EECS 210), which I also teach. Previously, I taught this course in a traditional lecture format (Fall 2016 and Spring 2017). In the Fall of 2019, I will teach it using a hybrid active learning-lecture format. Discrete Structures is typically a second-year course in the EECS major, which covers topics including logic, sets and functions, general proof techniques, mathematical induction, sequences and summations, number theory, basic and advanced counting techniques, solution of recurrence relations, equivalence relations, partial order relations, lattices, graphs and trees, algorithmic complexity, and algorithm design and analysis. Additionally, in collaboration with the KU Office of Institutional Research, we plan to track downstream performance of those cohorts of students who completed EECS 140/141 before and after the course transformation to a fully-flipped design.

Reflection from Dr. Johnson:

When I first began the process of transforming EECS 140/141 to a fully flipped class, my immediate concern was, “Will this work?” I was not sure how students would perceive a fully flipped course that was so different from the traditional lecture courses that I had previously taught. On the first day of class, I asked students, who were all incoming freshman, whether they had any experience in a fully flipped course. About 10% of the students raised their hands. As instructors, we may think that students will be resistant to modern teaching techniques; however, there are quite a few middle schools and high schools that have implemented active learning and flipped classroom designs. The truth is that students are probably more used to this idea than we think, and that this experience will only continue to grow over time.

My role as an instructor has changed considerably in this new, flipped classroom environment. I now spend no time standing in the front of the class and lecturing to students. However, it is important to note that less time lecturing does not mean less time teaching. In fact, I find my job as an instructor in a flipped classroom much more difficult than my job as a lecturer. I also feel that I am better able to understand student challenges and the ways in which students engage with course material, which helps me to tailor class materials to ensure that students will come to class prepared with the appropriate information that they need to succeed.

When I first started to incorporate active learning using in-class problems in Fall 2017, I took a passive role as a facilitator, standing and watching for raised hands and addressing specific student questions as they came up. I later realized that when I walked around and talked to groups without being asked, students were less hesitant to ask me questions. Now, I spend every minute of class conversing with students and answering questions as I walk from group to group. While this can be somewhat exhausting, I also leave the classroom thinking that I have actually helped students to understand something, which makes for more effective student learning and more satisfying teaching. In the fully-flipped class model, it is easy for me to get a sense of the level of difficulty of the problem sets, the processes that students undertake when trying to solve an unfamiliar problem, and common (and uncommon) misunderstandings, all of which might remain mysteries to me if I were lecturing instead of interacting with students. Similarly, I can tell that my UGTFs also get a lot of satisfaction out of their participation in this course. As part of a CTE Course Transformation Grant, I was able to hire two previous UGTFs to develop in-class problems for this course. Now, the current UGTFs are responsible for helping to facilitate the in-class work, while the GTAs lead lab sections and grade in-class problems. We think that including UGTFs and GTAs in course redesign represents a model of peer-to-peer learning that makes these large transformations sustainable.

I will likely be teaching EECS 140/141 every semester in the near future. This consistency provides a valuable opportunity for me to gradually tweak small aspects of the course to improve student learning each semester. I am incredibly grateful for the assistance of Dr. Molly McVey, who gave me the initial idea to transform my traditional lecture-style course to incorporate active learning beginning in Fall 2017, and who encouraged me to apply for the CTE C21 Course Transformation Grant that ultimately allowed me to fully flip my course beginning in Spring 2018. Dr. McVey continues to be instrumental to the course transformation process by helping to collect and analyze data pertaining to long-term performance of students in subsequent EECS courses (retention of introductory material) and DFW rates among several semesters of EECS majors, especially to assess the effect of teaching style on the success and retention of under-represented groups in engineering.

In summary, student performance following the fully-flipped implementation of EECS 140/141 lent support to other studies that have found that student performance is greatly improved with active learning practices (Felder et al. 2000, Freeman et al. 2014), especially concerning non-white-male students. While implementing a fully-flipped course required a lot of initial preparation work (e.g., designing problem set modules, creating question banks for Blackboard quizzes, and preparing video lectures and narrated PowerPoint slides), I think that the effort was worth it. Data suggest that students in EECS 140/141 are now more engaged, and show improved achievements of course learning goals. Plus, we have a more satisfied instructional team.

I hope that the following major takeaways might help other instructors who consider implementing a flipped classroom design:

• Make your classroom space work for how you want to teach. While a setup of group tables with plenty of space for facilitators to walk around seems ideal, an auditorium-style classroom has worked just as well, or better, for our flipped-style classroom, since raised tiers make it easier to see and address students. Informal, cooperative groups, as opposed to team-based learning, can allow for more flexibility in classroom setup.
• Grading in-class problems takes just the same amount of time as grading homework problems. Prioritizing in-class learning experiences for students will not necessarily correlate with more time grading.
• I put a considerable amount of time into transforming the EECS 140/141 classroom upfront, but the majority of this work will not need to be re-done every semester. For example, designing question banks for online Blackboard quizzes was very time consuming, but now I can continue to pull from these questions, and all of the quiz grading is done automatically, online, which saves a lot of time in the long run.
•  

Reflection from Dr. McVey:

Being a part of the course transformation with EECS 140/141 professor Dr. Johnson has been really inspiring and rewarding. When we began working together, Dr. Johnson was interested in the active learning/flipped classroom model, but hesitant to bite off too much at once. After his experience with the summer camp, he decided to jump in to active learning; that new enthusiasm and motivation to transform the course was helpful in moving along so quickly. I think that it does take a “lightbulb” moment for instructors to realize the impact of moving to active learning techniques—it is hard to get the momentum going if they don’t have a clear “why” in their mind for why they are taking on the initial time commitment to transform their course. Once that experience happens where they see the impact, it becomes much easier because they know what they are working towards and why.

Another thing I will say is that this effort was truly a collaboration between Dr. Johnson and myself, and this is part of the reason I think the postdoctoral researcher model is so effective. I provided some of the background and initial motivation, connected him with some internal resources (CTE course transformation grants, CODL resources, etc.), and consulted with him often as he was working through the process, but Dr. Johnson was the one driving the transformation. Then, as he implemented this model, I was able to help with collecting student learning data and survey data, and after each exam we analyzed the modules that seemed to be presenting problems for the students. We also celebrated the big improvements we were seeing in student learning. This model of collaboration was effective and efficient; with a small investment of my time and work, Dr. Johnson was able to take off and has implemented an extremely successful version of a flipped classroom. The results speak for themselves!

Acknowledgments:

This material is based upon work supported by the National Science Foundation under Grant Number DUE1525775. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.