Improving Comprehension and Application of Communication Research Methods—Jeffrey Hall (2009)
A communication studies professor sees positive changes in student learning in a restructured, upper-level research methods course.
COMS 356: Introduction to Behavioral Research Methods is a required course for all upper-level communication studies majors; it is both an introduction to methods and an introduction to statistics. After teaching this course once in Fall 2008, I felt that the lessons, material, and project could be better aligned with student needs and abilities and could, overall, be more efficient. I redesigned the course in order to maximize students’ learning and to help prepare them for professional work.
In Fall 2009 I implemented three main changes:
- Provided practice exams, problems, and other learning resources.
- Used lecture times more efficiently, focusing on particularly important information rather than including a significant amount of “nice to know” information, and did more activities and practice problems in-class.
- Revised course assignments and the final project.
I assessed student learning and the impact of these changes through work done in weekly lab sessions, performance on traditional exams, and the completed final project.
The overall grade distribution for the course was similar to that in prior years, although the scaffolding I implemented seemed to positively affect the final project scores. Although students appeared to better understand course material, there were areas that still needed improvement. Students still struggled with making research-based arguments when writing hypotheses, choosing the correct test for their question, and using statistical analyses correctly. They also would have benefited from having more practice in interpreting statistical information.
Overall, my impression of the changes implemented was very positive; I achieved my goals and also saw positive changes in student performance. I will maintain the changes, extend scaffolding, and continue to focus lectures and clarify assignments, as well as consider how to further modify the course and introduce other structural changes in future semesters.
^Back to top^
Introduction to Behavioral Research Methods (COMS 356) (pdf) is a required course for all communication studies majors. In the past, students often took it in their final year of coursework. The COMS department increased the number of sections to allow students to take the course in their junior year so they will be more prepared for upper level courses, particularly courses reporting on social scientific research that may require students to use those methods for class projects. Typically, 80 students take the course each semester. However, to clear backlogged majors, two courses of 55 students each were taught in the Fall 2009 and Spring 2010 semesters. The course is comprised of two 50 minute lectures and one 50 minute lab section each week.
COMS 356 is both an introduction to methods and an introduction to statistics. It is designed to teach basic empirical and social scientific methods, to provide students tools to design their own projects, to help students learn to interpret statistical results, and to provide them with a basic understanding of software used to analyze both survey and experimental data. To successfully complete the course students must be able to demonstrate their understanding of the following concepts: measurement validity and reliability, hypothesis testing, operationalization, survey design, experiment design, and common statistical analyses such as a t-test, correlation analysis, and ANOVA. In addition, improved critical thinking, social scientific writing, and ethical data interpretation and analysis are the course’s overarching goals. To improve ethical interpretation of messages, I try to teach students to differentiate between significant and meaningful effects, identify how research designs influence the accuracy of results, and present data accurately and honestly. The course should prepare students for professional work in marketing, public relations, or sales, careers that will require them to measure message or communication intervention effectiveness and also understand statistical relationships so as to ethically interpret message effects.
This course is challenging in several ways. Students struggle with the exams because they have difficulty transferring knowledge from lecture notes to practice. Also, because it is a required methods course, traditionally taken by seniors, there are additional pressures both on and from students due to the complex curriculum and upcoming graduation deadlines. Finally, the scope of course topics make this demanding to teach, and I wanted to make it more efficient.
I was not satisfied with my first experience teaching the course; I felt that the class lessons, material, and project could be better aligned with the expectations and abilities of most of the students. Because I will teach this class at least yearly, I want to approach it in a way that maximizes student learning. This led me to ask: how can I improve COMS 356 student learning?
^Back to top^
In Fall 2009, I attempted to address some of these challenges. First, to help students be more prepared for exams, I created practice exams for them to download for credit, and I refined the course content to reduce the amount of material covered. Second, to improve knowledge transfer, I provided and worked through more in-class examples. Third, to use instructor and GTA time more efficiently, I refined and focused the project assignments. Finally, I focused on maintaining a positive attitude and building rapport in the class by slowing down the pace of the course, being more patient with struggling students, and reminding students that I was committed to their learning.
- Practice exams
Utilizing the COMS 356 Blackboard site, I supplied statistical problem examples, as well as quizzes and practice problems (pdf) very similar to those on the actual exam. In addition, I made a greater variety of learning resources available to students, including handouts, practice problems, examples of good final projects (pdf), APA formatting, and other materials.
- In-class changes (lectures and examples)
The first third of the course is dedicated to introducing the scientific method and understanding research questions, measurement operationalization, and reliability and validity. To improve this section, I focused on conveying information that was particularly important for completing the class project, rather than including extra “nice to know” information. Additionally, the operationalization lecture, which should flow into the independent variable, dependent variable, and levels of measurement lecture, was extended and improved dramatically. The improved lecture utilized a fun activity to listen to songs about love and write items to measure love as expressed by the singer. In the next third of the course, I focused on survey and experimental design components, matching the progress students were making on their own projects. The final third of the course incorporated many more practice problems completed at the beginning of each class meeting.
- Project assignments
When I taught COMS 356 in Fall 2008, I gave six assignments, but I was not satisfied with the first three. The second three assignments covered Excel data entry, descriptive statistics, and hypotheses testing. These were mainly exercises in SPSS, and there was a substantial amount of class and lab time devoted to statistical interpretation. For Fall 2009 I rethought the first three assignments and the lectures leading up to those assignments and devoted more class time to ensuring students were properly prepared to complete them.
I also revised the final project. In previous semesters groups choose their topics, which proved to be time consuming; identifying appropriate articles for each groups’ project was problematic and required much GTA time. These two issues made it very difficult for students to complete a literature review that could lead to meaningful hypotheses. To make it easier and more efficient for both students and my GTAs, in the revised assignment I restricted the students to four possible project topic options, identified the independent and dependent variable and the measurement of each, and provided one research article on each topic.
Opportunities to demonstrate understanding
I used three methods to assess student learning and the effects my changes had on that learning. First, the students had weekly GTA-led lab sessions primarily for the purpose of developing their course projects. Each week groups of two or three students worked towards completing those projects and the GTA collected examples of their project development. Second, students took three exams during the semester. Their performance on those exams helped me to reflect on what students had and had not learned. Third, the students handed in their final project, which represented a cumulative example of their learning in the course overall.
^Back to top^
The overall grade distribution was similar to prior years, but with more B and B+ grades. This distribution indicated greater student success as none failed the course (in comparison to three F grades in a previous semester). While there were students who failed the tests but passed the class, this may be a consequence of reallocating points to the assignments and final paper.
Generally the final paper grades were high because their content was so carefully scaffolded throughout the semester. This was positive in that students had a better final product, with students more consistently turning in correctly compiled papers with all requirements.
Overall, I am confident that students had a better understanding of the course material than did students in previous semesters. I think substantial improvement could be seen in operationalization, in interpreting degrees of freedom and the relationship between t values and p levels, and in knowing what steps to take to initiate the correct statistical analysis to practice problems. However, students still seemed to have problems synthesizing course notes and information for both tests and the final paper. I feel that there is a disconnect between lecture and lab, where most of the project work is conducted, and more effort is needed to make connections between lecture and class projects.
There are several areas where I see room for improvement. There are a few very important areas where students should demonstrate competency on the final: 1. hypotheses resulting from making a research-based argument, 2. what do statistical tests mean in relation to a hypothesis, and 3. what do the tests mean in relation to hypothesis testing.
First, students appeared to struggle with making a research-based argument when writing hypotheses. This starts with conducting a purposeful literature review. Students often focused too much on the reviewed studies’ details rather than the research question and analysis. In addition, students struggled with moving from argument to hypothesis, and I think more activities are needed that help students learn the difference.
Second, students did not always use statistical analyses correctly or use the correct test for their research question. Some students offered an incorrect interpretation of results. (I have chosen to provide only a few pages of each paper to serve as an example instead of the full paper.) For example, Student A (pdf) reports a significant negative correlation for the second hypothesis, but in an unexpected direction. The student did not correctly report significance, but did correctly report support for the hypothesis. Some students did not report the correct statistics. Student B (pdf) not only reports a sex difference that is not part of the final paper assignment, but also reports mean differences in a way that implies a statistical difference. In addition, Student B shows a common problem of reporting the difference in ad sponsor (Tanqueray vs. Belvedere) rather than a difference between the explicitness of the “drink responsibly” message, which is the independent variable. Some students successfully completed the assignment and ran the statistical test. Student C (pdf) reports statistical results accurately and in relation to the hypotheses. In order for all students to successfully use and interpret statistical tests in relation to the hypotheses on future final papers and exams, additional instructor and GTA effort may be needed.
Finally, I think that more effort might be made to practice explaining the results in terms of the data. Once students analyze data and examine the appropriate statistical tests for significance, they need more practice in interpreting statistical information in relation to their hypotheses and measurement. This may be a skill that can be practiced more in their labs.
^Back to top^
I saw positive changes in student performance on assignments, final projects, and exams. Changes in quantitative student evaluations and written comments also suggest that the modifications I made to the course this semester led to improved learning. I am most pleased that students reported their perception that I was dedicated to student learning. I think changing my style of teaching really paid off in that regard. While it may be that a smaller class size contributed to improved learning for students, these evaluations and student performance still provide evidence of change in the right direction.
While students performed better than they had in Fall 2008, both in terms of final grade and final project, there are still areas of concern. Students still struggled with some key concepts, and additional supports may be needed to get them from “going through the motions” to applying ideas.
In evaluations, many students said there was not a clear link between lab activities and lecture. Therefore, more work is needed for clarification, and I would like to find ways to help students better understand that link. I am currently considering ways to emphasize that connection.
I am considering several other modifications to the class. First, working with the KU Library and Instructional Development and Support (IDS), I am going to integrate the use of library resources for the first assignment. This should help students identify useful and topical articles. I will work with librarians to see if they can assist with some required training that students would attend the week before their first assignments are due. Second, I am going to try to work with IDS to see if I can build a student SPSS training session into the class.
I would also like to add a lecture or library presentation on literature reviews, a session such as “Literature review: how to write it, what it is for, and how to read an article.” I may need to invite someone from the library to help design and execute this lecture or lab session and to write assignments that best capture student learning. My impression is that many students use the first article they find on a topic without considering its relevance to their project. Students need to learn what a research article is and receive help in determining whether a specific one applies to their project. More time should be spent on how to write these reviews, with students possibly looking at both strong and weak examples.
There are a couple of further structural changes I might make the next time I teach the course. First, I want to find ways to organize class sessions so that students are more active in answering questions and using peer feedback on their work. Second, I will probably make assignments due on Friday rather than Monday to encourage students to use GTA office hours while completing assignments, rather than emailing on Sunday night. This will also give the GTAs more response time. Third, I may add more basic questions involving definitions and material straight from lectures to the final to give students some early and easy successes before attempting the more challenging and application-based material. Fourth, I am considering making my PowerPoint lectures available to students via Blackboard so they have access to them outside of the class period. Student comments indicated they wanted access to both lecture notes and more examples, and I am thinking about the value of providing them. Finally, I may go back to taking and counting attendance, as I saw lower lecture attendance numbers during the semester.
Overall, my impression of the changes I made was very positive. I achieved my goals of restructuring projects, clarifying expectations for those projects, reducing instructor and GTA output in helping students choose a project, and helping students to gradually and effectively take advantage of built-in feedback mechanisms. I think these changes contributed to my teaching by helping me to organize my lectures, trim extraneous information, and change my general attitude about what I am there to accomplish as a teacher. I am trying to adopt a position of being a facilitator and trainer. Inspiration and personal excitement about teaching can go only so far for many students. I will maintain all the changes I made this semester, extend the scaffolding, and continue to focus lectures and clarify assignments. All of these changes were very useful and should continue to improve students’ learning, as I continue to refine and improve the course.
Contact CTE with comments on this portfolio: firstname.lastname@example.org.
^Back to top^