Building KU's Teaching and Learning Community

Using Multiple Evaluation Methods to Assess Undergraduate Learning in the School of Journalism and Mass Communications (2009)

Video cameraOverview

Faculty members in the William Allen White School of Journalism and Mass Communications have used their re-accreditation opportunity to assess and document student learning and evaluate how to better fulfill their values and mission. Through this process, they have developed and implemented rubrics in core courses, pre- and post-tests, real-client feedback on student coursework, and student exit surveys.

Background

Nationally accredited since 1948, the William Allen White School of Journalism and Mass Communications currently enrolls more than 700 undergraduate students and more than 60 graduate students. The School prides itself on an innovative and evolving curriculum that teaches traditional values and skills while embracing a world of rapidly changing technology. The School’s values statement, mission statement, and the professional values and competencies specified by the School’s accrediting body served as the sources for the objectives and specified outcomes that shaped the School’s learning goals and subsequent assessment tools.

Implementation

Faculty members have found that using rubrics to evaluate student work on comprehensive projects has helped professors as well as students identify particular strengths and weaknesses in specific assignments. From the grammar pre-test data, professors have identified high, medium, and low areas of need. In the capstone course, real-world clients share their input with students and faculty on the quality of the services delivered and campaign presented, including feasibility and the degree to which the campaign met their needs. In exit questionnaires, graduating seniors evaluate their experience and skills. In one recent survey, about half of the students agreed that the School met their expectations, with 25% of the seniors stating that the School exceeded their expectations. Concomitantly, more than 50% of the graduating seniors strongly agreed that they would be competitive in the job market.

Student Performance

The assessment process has beneficially increased the amount of communication and collaboration among professors of complementary courses and is benefiting the School’s ongoing curriculum-revision process. The faculty has realized that the assessment plan is a work-in-progress and has found that some measures that initially seemed reasonable were not, while other measures have emerged through the increased focus on assessing student learning. The faculty has found that coordinated, School-wide assessment of student learning requires that the process be integrated into School governance so that a committee can oversee and ensure implementation of the plan.

Reflections

Faculty members in the William Allen White School of Journalism and Mass Communications have used their re-accreditation opportunity to assess and document student learning and evaluate how to better fulfill their values and mission. Through this process, they have developed and implemented rubrics in core courses, pre- and post-tests, real-client feedback on student coursework, and student exit surveys.


^Back to top^

stack of newspapersBackground

Nationally accredited since 1948, the William Allen White School of Journalism and Mass Communications currently enrolls more than 700 undergraduate students and more than 60 graduate students. Upon graduation, students in the News and Information track are prepared for careers in a variety of media, including newspapers, magazines, broadcasting, and converged, online venues. Students in the Strategic Communication track are prepared for careers in advertising, public relations, promotions, and sales and/or media management. The School prides itself on an innovative and evolving curriculum that teaches traditional values and skills while embracing a world of rapidly changing technology.

The School has a strong commitment to student learning and student success. In 2009, for the third consecutive year, KU journalism students placed first in the national Hearst Foundation Writing Awards program, also known as the “College Pulitzers.”

Three sources of objectives and specified outcomes have shaped the School’s learning goals and subsequent assessment tools:

The School’s Values Statement, adopted by a unanimous vote of the faculty on Nov. 7, 1997

  • A diverse, collaborative and dynamic student-centered environment
  • Excellence in learning, teaching and mentoring
  • Free expression and conscientious, ethical journalism as cornerstones of a democratic society
  • Critical and creative thinking
  • Meaningful research and creative activity
  • Imaginative outreach and collaboration on this campus, in this state and in the professions of journalism

The School’s Mission Statement, adopted by a unanimous vote of the faculty on Sept. 3, 1998

  • Appreciate the value of freedom of expression and its importance in society
  • Be able to critically analyze mass media
  • Have an ethical framework for the practice of journalism and mass communication
  • Demonstrate an understanding of the value of a diverse society, a diverse work place and the importance of reflecting that diversity in mass media
  • Demonstrate an understanding of the impact of changing media technologies on society
  • Demonstrate effective production and presentation skills for the media
  • Demonstrate an understanding of the importance of research for adding new knowledge and an ability to employ research techniques for analysis and interpretation
  • Demonstrate an understanding of how communications organizations function and the important management issues they face

The Professional Values and Competencies specified by the School’s accrediting body, the Accrediting Council on Education in Journalism and Mass Communications

The Accrediting Council on Education in Journalism and Mass Communications requires that, irrespective of their particular specialization, all graduates should be aware of certain core values and competencies and be able to:

  1. Understand and apply the principles and laws of freedom of speech and press, including the right to dissent, to monitor and criticize power, and to assemble ad petition for redress of grievances
  2. Demonstrate an understanding of the history and role of professionals and institutions in shaping communications
  3. Demonstrate an understanding of the diversity of groups in a global society in relationship to communications
  4. Understand concepts and apply theories in the use and presentation of images and information
  5. Demonstrate an understanding of professional ethical principles and work ethically in pursuit of truth, accuracy, fairness and diversity
  6. Think critically, creatively and independently
  7. Conduct research and evaluate information by methods appropriate for the communications professions, audiences and purposes they serve
  8. Write correctly and clearly in forms and styles appropriate for the communications professions, audiences and purposes they serve
  9. Critically evaluate their own work and that of others for accuracy and fairness, clarity, appropriate style and grammatical correctness
  10. Apply basic numerical and statistical concepts
  11. Apply tools and technologies appropriate for the communications professions in which they work

^Back to top^

press passImplementation

In order to fulfill values and mission, meet accreditation standards, and better assess and document student learning, the School unanimously adopted a Plan for Assessment of Student Learning (pdf) in 2008. As the plan has been implemented, it has evolved to include four core elements: assignment rubrics for core comprehensive projects; pre- and post- surveys and testing on subjects including grammar, media law, ethics, and the value of diversity; feedback from external clients on student capstone coursework; and exit interviews with graduating seniors.

Assignment rubrics
Professors have designed assignment rubrics in collaboration with the Center for Teaching Excellence; the rubrics help identify and measure skills that students must master. These rubrics are applied to a radio commercial assignment for students in Jour 435: Message Development. The assignment has multiple parts that require students to develop a strategic message planner (a document that consolidates research and identifies one, clear, strategic message for the commercial), a production script, and an audio production of the script. Similar rubrics were developed for a more comprehensive, multimedia integrated advertising project (IAP).

Pre- and post-surveys and testing
Pre- and post-surveys and testing covering subjects including grammar, media law, ethics, and the value of diversity are given to students at the beginning of their School careers and then again after key coursework.

Every semester, the School surveys students in Jour 101: Media and Society to gauge their knowledge of media ethics and law. A School committee developed the 35-question multiple-choice questionnaire, based on an early draft by Trevor Brown, former dean at Indiana University. The goal is to gather baseline information on what students know before they enter the School of Journalism and then administer the same questionnaire after students have completed courses in the First Amendment and ethics. Of the 450 students enrolled in the class in the Fall 2006 semester, 350 took the survey for three extra credit points on one of their assignments.

Every semester, students take a grammar test that covers punctuation, vocabulary, syntax, and parts of speech. Students who fail to pass the test must enroll in Jour 002: Grammar and Usage before proceeding to other School courses. For more information, see our complete list of topics/areas covered by the grammar and usage test.

Feedback from external clients
In the capstone course Jour 676: Strategic Campaigns, students create research-based strategic communications campaigns for real-world clients. The group project requires that students ultimately present their campaign to their real-world clients. The clients provide feedback on student performance (and the services delivered and campaign presented, including real-world feasibility and the degree to which the campaign met their needs) through a written evaluation form and post-presentation remarks.

Exit interviews
Graduating seniors respond to an exit questionnaire that asks them to evaluate their experience in the School in the following areas: writing skills, research abilities and skills, communication skills, career preparedness and job market competitiveness, the importance and their understanding of the First Amendment and ethical issues, their understanding of the role of diversity in society, the classroom instruction as it reflects current trends in communications technologies, the access to and instruction in modern mass communications technologies, academic advising, faculty friendliness and approachability, faculty mentoring, adequacy of hands-on experience and theory instruction, availability and usefulness of internship opportunities, and the advising center. The exit questionnaire also has a section for additional comments.


^Back to top^

headphones and radio microphoneStudent Work

Assignment rubrics
The assignment rubrics developed are used to grade student work. The commercial assignment (pdf) consists of rubrics (pdf) for a strategic message planner, a production script, and a production of the script. Specific elements as well as overall assignments were designated as Professional quality, Competent quality and Incomplete quality. Descriptions of each of these categories can be found on the commercial assignment rubric, linked above.

The following are examples of graded student work from each of the three categories.

After using the rubrics to evaluate student performances, a group of professors met to discuss what they learned about student performance from the rubrics and how they could improve instruction, assignments, and the rubrics. They also identified the mean scores in each rubric category (pdf) and identified the areas of overall weakest performance.

Findings:

  • The rubrics helped professors as well as students identify particular strengths and weaknesses in student performance.
  • The rubrics helped professors remember to check all the core aspects of complex assignments.
  • Students seemed to readily understand the forms. They had few questions about grades.
  • Early versions of the rubrics needed (and have received) refinement. They covered the tactical/procedural parts of the assignments, but they did not adequately cover the elements of creativity inherent in the assignments.

Next steps:

  • Continue to refine the rubrics, particularly as weaknesses become apparent and assignments evolve.
  • Expand the use of rubrics to other assignments in other courses.
  • Build library of rubrics and actual assignments that earned professional, competent, and incomplete ratings.
  • Use library of evaluated student work, plus records of how each class is doing, to help assess student learning.
  • Use library to help new teachers who may be uncertain about grading standards and amount of feedback given.

Both the commercial assignment and the integrated advertising project (IAP) results have been shared at faculty meetings and, particularly, with professors who will be teaching Jour 435 (and those assignments) in the upcoming semester. So far, evidence from the rubrics has helped Jour 435 professors pinpoint which parts of those complex assignments need additional explanation. Because of evidence discovered in the Documenting Learning Success project, Jour 435 professors currently are testing a new way to organize and teach a core aspect of the IAP project.

Pre/post surveys and testing on subjects including grammar, media law, ethics, and the value of diversity

Grammar

In one recent semester, 122 students took the grammar test. The number of incorrect responses for each area of grammar and usage was tallied and split into areas of need: high (missed by 44–122 students), medium (missed by 24–43) and low (missed by 3–22).

  • High-Need Areas: problems with plurals occur in three categories: punctuation (possessives), vocabulary (freshman/freshmen) and pronouns (its/their). Click for a more in-depth description of the Top Five Grammar Problems identified in a recent test.
  • Medium-Need Areas: commas, vocabulary (more than/over, fewer/less), passive voice, and pronoun case.
  • Low-Need Areas (problems that occur less often but merit review): quotation punctuation, colons/semicolons, vocabulary (principle/principal, affect/effect, try and/try to, sit/set), dangling modifiers, parallel construction, and pronouns (whose/who’s, that/who).

Click for a complete analysis of a recent grammar test results.

As a result of such analyses, Jour 002: Grammar and Usage professors have been able to tell students which concepts their peers from the previous semester found the most difficult. The evidence has also reshaped lesson plans; for example, professors spend more time explaining pronoun case in elliptical clauses: She is taller than I.

Grammar results have been shared with faculty via the e-mail list serve and in faculty meetings. Informal feedback suggests faculty members are grateful for the explanations that accompany the identification of the students’ most-common grammatical errors.

Media law/ethics
From the 350 student responses to the ethics pretest from students in Jour 101 in a recent semester, professors found that:

  • The majority of students—roughly 2/3—have a good knowledge of the First Amendment and libel. The exception is obscenity. Most students believe the First Amendment protects obscene speech.
  • The majority of students—roughly 3/4—say they would be “very concerned” or “somewhat concerned” if fellow classmates engaged in unethical behavior, such as cheated on a test. However, of that percentage, the majority falls into the “somewhat concerned” category. An issue to consider is how to move them to the “very concerned” category.
  • The majority says it would be “very concerned” or “somewhat concerned” if a journalist or strategic communications professional plagiarized or fabricated information. Again, an issue is how to move students from “somewhat concerned” to “very concerned.”
  • Fewer than half the students have a good understanding of what ethics is, the Potter Box, and ethical frameworks.

The semester these surveys were administered in J101, the professor included a session on ethics and the First Amendment early on in the semester, before the surveys were completed; this may have affected student knowledge.

Next steps
The School will try to run cross-tabs to see if there are differences between men’s responses and women’s responses, or responses of news and information majors vs. strategic communication majors.

Feedback from external clients
External clients provide feedback in capstone courses such as Strategic Campaigns (Jour 676), in which students create research-based strategic communications campaigns for real-world clients and make presentations to those clients. Clients provide feedback on student performance, including feasibility and the services delivered, through a written evaluation form and post-presentation remarks.

The following is a selection of feedback from real-world clients.

Client survey questionnaires:

Client feedback videos:

Coca-Cola (2002): We’ve worked twice for Coca-Cola—once when the Coke contract was new at KU in 1999, and again three years later. This campaign became a model for what other Coke campuses can do.

 

 

Sprint (2005): We’ve produced campaigns for Sprint, the region’s largest employer in 2002, 2005, and 2006. This 2005 campaign improved Sprint’s image in Kansas City prior to the Sprint Center opening.

 

 

Commerce Bank (2005): The region’s largest bank serves approximately one-third of the KU student body and needed help in retaining graduates as future bank customers in the years following graduation.

 

 

After the Fall 2008 semester, six professors of Strategic Campaigns (Jour 676), informed by feedback from recent real-world clients, met to discuss student performance for the semester.

The following observations were made:

  • Grading Jour 676 is difficult: Because the course involves real clients, student nonperformance or substandard work isn’t an option. Professors and group members sometimes coach substandard performers up to the necessary level.
  • Students lacked problem-solving skills; many still wanted a checklist, a set of instructions.
  • Students often wanted to see plan books from previous semesters to use them as models.
  • However, students ultimately did deliver high-quality work that pleased clients.
  • Specific strengths were information gathering (particularly in assessing the problem/opportunity), responsibility, student realization that they needed to solve the problem, and multimedia execution of tactics.
  • Copywriting and design skills were adequate—with room for improvement.
  • Some of the multimedia tactics were praiseworthy, but students too often were learning the requisite software skills on their own. To improve this, the School could coordinate ways to make such knowledge available and then emphasize the need to gain that knowledge.
  • There was general consensus that the school offered little instruction in presentation skills.
  • There was strong consensus that, in Jour 676, students greatly increased their ability to work in groups—to learn from one another and, as a group, to generate creative ideas.

The conversation resulted in the following suggestions:

  • Increase student knowledge of and experience with critical-thinking and problem-solving skills. This should occur across the curriculum.
  • Coordinate ways to help students gain knowledge of research/communications software and emphasize the need to gain such knowledge.
  • Seek opportunities to increase student experience with presenting, copywriting and design/layout.

Exit interviews with graduating seniors
The data collected from the exit interviews delivers a good image of the achievements of the School viewed through the eyes of graduating seniors and other students. In one recent semester, 311 students responded to the exit survey. Of those, 112 were graduating seniors. Faculty compared responses from graduating seniors with responses from other undergraduate students. From the findings, the School has determined that although it seems to be doing a good job, there is room for improvement. For more information, please see a complete list of the student responses to each question divided by cohort can be found.

Observations made based on the analysis of student responses:

  • Research: There is an evident difference in perception of the ability to conduct research between seniors and other undergraduates. Completion of Jour 676: Strategic Campaigns may have something to do with this.
  • Ethics: Other undergraduates seem to understand ethical and First Amendment issues better than seniors. This may be because of recent curricular improvements to which seniors were not exposed.
  • Diversity and society: The findings resembled those of the ethics studies noted immediately above. The School therefore hopes to see improvement of this measurement for graduating seniors in the near future.
  • Access/Instruction in modern mass communications technologies: Although it may be an age-related issue, the data suggest that School’s efforts to provide access to modern technologies from both a practical and instructional perspective are successful, although less so for seniors compared with next cohorts.
  • Advising: The School does a good job advising all undergraduates and providing information about internships and job positions. At the same time, seniors tend to give increased importance to their internship experience as valuable for their future careers compared with other undergraduates.
  • Faculty friendliness and approachability: More than 90% of students consider the faculty friendly and approachable; they report that faculty members do a good job of mentoring them (of course, there is always room for improvement).
  • Preparing for a future career: About half of the students agree that the School met their expectations, with 25% of seniors stating that the School exceeded their expectations. There is room for improvement in this area, especially in providing students with hands-on experience. Concomitantly, over 50% of School graduates strongly agree that they will be competitive in the job market.

The open-ended questions provided insight into the needs and wants of School students. Some of the most frequent comments were:

  • Students enjoy Strategic Campaigns, but they believe the workload is excessive.
  • There is an obvious need for instruction in visual communication, both theoretical and practical.
  • New media, new media, new media: There is a perception the School focuses mainly on print/TV/radio and ignores current trends in communication channels.
  • Some students perceive the School puts too much emphasis on ethics and First Amendment and that semester-long courses are too long.
  • Students believe they need more classes that stimulate/channel their creativity.

^Back to top^

press passReflections

As the faculty reflects on its concerted efforts to assess and document student learning, several shared beliefs have emerged:

  • The School’s Plan for Assessment of Student Learning is a work in progress: Some measures that seemed reasonable are not; other measures, not included in the plan, have emerged through the School’s increasing focus on assessing student learning. For example, developing a common “client feedback form” for the many sections of the Strategic Campaigns course—each with a different client—became cumbersome; good information is coming from the individual forms. Also, as the faculty began to compile and discuss student-learning data, we began to develop an additional form of assessment: a panel of highly qualified professionals who could meet to review and assess selected final projects from students.

  • The assessment process has beneficially increased the amount of communication and collaboration among professors of complementary courses.
  • In the writing courses, the assessment process is helping to pinpoint specific areas for improvement in critical/creative thinking and grammar/syntax.
  • Assessment of student learning is benefiting the School’s ongoing curriculum-revision processes.
  • Coordinated, School-wide assessment of student learning cannot happen without the processes being integrated into School governance; a committee must oversee and ensure implementation of the plan.

Contact CTE with comments on this portfolio: cte@ku.edu


^Back to top^

News & Notes

See how CTE is leading the shift to flexible teaching at KU.

Visit the Flex Teaching sitewhich provides help for creating flexible courses that can shift between in-person and online.

Portfolio & Poster Search



List of all portfolios & posters