Building KU's Teaching and Learning Community

Combining Live Performance and Traditional Assessments to Document Learning—School of Pharmacy

Overview

The KU School of Pharmacy is combining live performance assessment with its traditional core science assessment, all in the same exam, and presenting the results for decision-makers to use as evidence of students’ preparedness for professional practice.

Background

Documentation of teaching outcomes is now commonly expected in academic accreditation self-studies, especially of professional schools that are examined by professional practice bodies as part of their effort to develop their field. Outcomes defined beyond the teaching setting where professional practice occurs are increasingly being added to traditional means and are also expected to also drive curriculum and assessment. Pharmacy is one of those professions with its accreditation standards defined and monitored by a body of professional practitioners, the Accreditation Council for Pharmacy Education (ACPE). Thus, after two years of development, the University of Kansas School of Pharmacy is combining live performance assessment with its traditional core science assessment, all in the same exam, and presenting the results for decision-makers to use as evidence to justify why particular students should or should not be passed on to professional practice.

Implementation

The practice-based portion of Pharmacy's criterion assessment exam consists of standardized clients in 12 testing stations representing different clinical or management problems. Difficulty is scaled to each student's level of preparation. The cases standardized clients represent are patterned from post-college pharmacy practice with the involvement of practicing pharmacists. They are constructed to be as authentic as possible in simulation, and to be valid and reliable by the highest assessment standards. The exam involves five steps:

  1. Prequel case where students answer a ten-item quiz based on a written description,
  2. Instructions which prepare students for the live-action encounter,
  3. Encounter with a standardized client,
  4. Ten-question quiz over the content and knowledge covered in the encounter, and
  5. Review period in which the standardized client leads the student to reflect on the encounter just concluded.
Student Work

Students receive feedback about exam results in two ways, one informational and the other experiential:

  1. Information in the form of individual summary scores are sent to individual students. They learn their results as a percent of all points available.
  2. The recent experience with a realistic client is the basis for a reflection discussion on what happened and could be improved.
Reflections

It is intended that the standardized client program will become a capstone experience within the School of Pharmacy; where students’ progress through the program is dependent on their successful demonstration of professional competency in both the scientific and clinical/counseling domains of practice. If the assessment can be refined to provide additional detailed information about where students struggle within the different domains of professional competency, then the School of Pharmacy can better prepare its students for their first clerkship and subsequent professional service in a fast-growing and ever-changing field.


^Back to top^

Background

Documentation of teaching outcomes is now commonly expected in academic accreditation self-studies, especially of professional schools that are examined by professional practice bodies as part of their effort to develop their field. Outcomes defined beyond the teaching setting where professional practice occurs are increasingly being added to traditional means and expected to also drive curriculum and assessment. Students are additionally expected to be directly observed as if their services were being delivered in practice. Obstacles that prevent a reliable, cost-effective student exam in the field are insurmountable, but a standardized client exam simulating live performance that can be staged in the more controllable professional school setting can now be accomplished reliably.

The new expectation that outcomes be documented certainly applies in the health professions such as medicine and nursing, where it has driven development of live performance assessment methodology using standardized clients for 40 years. The National Board of Medical Examiners has used live performance in its licensing exam since 2004. “Standardized” means clients represent certain cases essentially the same way with every student so the exam is virtually the same every time, which resolves a major criticism that exams with live performers are inherently inconsistent. Consistency can be built into each case that students will encounter. Live performance exams that have always been favored for their validity—because they are designed to be realistic—can now also be designed to be reliable, i.e., fair. This includes that standardized clients be recruited as employees and rigorously trained for the control this brings for the needed reliability.

Outcomes assessment is increasingly expected in non-health professions too, where application with a client must be shown, in addition to mastery of core sciences, such as in engineering, social work, business, and law. Now the professional school is expected to produce two types of data showing not only that (a) each student has achieved minimal understanding of the field’s core knowledge—the traditional way to assess competency—but also (b) the student can display the communication and interpersonal skills necessary to begin to successful practice in the profession—the part of competency assessment that is new and increasingly reliable.Now students are advanced to practice sites with defined practice competencies, as well as knowledge.

Pharmacy is one of those professions with its accreditation standards defined and monitored by a body of professional practitioners, the Accreditation Council for Pharmacy Education (ACPE). Thus, after two years of development, The University of Kansas School of Pharmacy is combining live performance assessment with its traditional core science assessment, all in the same exam, and presenting the results for decision-makers to use as evidence to justify why the particular student should or should not be advanced toward professional practice. (It goes without precise accounting of the data that accreditation bodies require such outcomes to be validly and reliably measured.) Core knowledge and performance assessment are established using live performance scenarios and, combined with traditional knowledge exams, are now used to assess a broader range of competence toward pharmacy practice. In the three academic years ending in 2007, 2008, and 2009, the school has built an administrative program to provide this exam and implemented it with all current PharmD students.


^Back to top^

Implementation

The practice-based portion of the Pharmacy School’s criterion assessment exam consists of standardized clients in 12 testing stations representing different clinical or management problems with the difficulty scaled to the student’s level of preparation.The cases that standardized clients represent are patterned from post-college pharmacy practice with the involvement of practicing pharmacists. It is constructed to be as authentic as can be accomplished in simulation, and also to be valid and reliable by the highest assessment standards. Workgroups of practicing pharmacists contribute to the writing cases used in the exam; standardized client performance and observation reliability are measured after administration of each four stations to one graduating class of students. Objective studies conducted of a 10% sample of video recordings compare both students’ and standardized clients’ performances with those of independent observers. Authenticity is also measured with post-exam student questionnaires and direct ratings of the videos.

Traditional exams, including styles with multiple-choice and short-answers, are combined with this non-traditional performance-based assessment. Four cases are presented to each student each of three academic years. Each case is embedded in the process of a 25-minute station. Each station consists of five computer-paced steps using the sequence shown next. (Students spend five minutes at each station across a total of four stations in one day, a total of 100 testing minutes.)

  1. Prequel. Presents a computer-presented case that foreshadows the medications and clinical situation that will be used in the standardized client encounter to follow; includes a ten-item quiz.
  2. Instructions. Provides a clinical setup of the live encounter to follow, sets expectations for the student, and provides needed data. Generally, instructions begin, “Greet the client and respond to the following situation….”
  3. Encounter. Puts the student into the clinical situation in which the case is to occur. A range of possible student responses to the case have been considered and the standardized client instructed in observing them when performed and subsequently recording them as described in the next interval. A standardized client authentically represents pharmacy cases in the roles of patient, caretaker, medical practitioner or employee, and in settings of a community pharmacy counter, a pharmacy office, or a hospital. For examples of these videos, please see below.
  4. Core Questions and Exam Scoring. Examines science knowledge required to provide care in the encounter using a ten-item quiz. Concurrently, the standardized client records observations from the encounter on a different computer, including yes/no whether the student did 10-18 professionally proficient case-specific acts, such as “asked about allergies,” and also including a rating on a five-item global assessment scale on communication dimensions such as “degree of focus, logic, and coherence.”
  5. Reflection. Since the “learnable moment” prompted by the exam is made available, in the final five minutes of the encounter the standardized client leads a discussion in which the student reflects on what happened. No client-generated feedback is provided except for how the standardized client experienced the student in his or her practitioner role. The standardized client avoids volunteering any specific checklist item to maintain exam security. Students are invited to role-play parts of the encounter they found unsatisfactory. Reflection even without specific answers from standardized clients has been highly popular with students. Please see below for the reflection video.

Administration necessary to field such an exam requires more than currently exists in most professional schools, because live performance assessment is quite different from what usually exists for teaching. It consists of ten unique components that either (a) require major input from the profession’s content, or (b) are unique to live performance assessment but don’t require much professional content and could be used in any professional school.

Ten components of a standardized client (SC) competency assessment program

(a) Three professional components (require major input of the profession):

  • Competencies. Defined by KU School of Pharmacy and ACPE.
  • Case development. Written, validated, and standards set by nine to 12 different volunteer practitioners.
  • Evaluation. Exam results validated and tabulated for school’s decision-makers.

(b) Seven administrative components (exclusive of professional input):

  • Staffing. Program administrator, standardized client (SC) trainer, exam floor manager, academic official.
  • Marketing. Satisfaction of program’s constituencies, i.e., to break even yearly.
  • Event logistics. Planning so all students receive all exams while on campus.
  • Recruiting. Work to hire part time, temporary SCs as itinerant educators who must have skills and experience in observing students, performing a client role, and leading a reflection discussion to the student’s benefit.
  • Training. Practice to represent their case, score students, and lead reflection.
  • Event management. Implementation of three exam events per academic year; e.g., managing the exposure of all 105 Pharmacy students, in waves of four students, with exposure to four unique professional encounters, across nine five-hour days (in the five five-minute steps described above).
  • Facility. Designed in a lab-type space to exemplify significant practice settings.

Videos of standard client encounters

Patient Video

 

 

Video of a community encounter

 

 

Video of a hospital encounter

 

 

Reflection from standardized client

 

 


^Back to top^

Student Work

Exam data are reported to the school’s Assessment Committee to demonstrate individual student outcomes. Outcomes reported include performance and knowledge. Performance is reported by summary scores across all 12 cases in each of four pharmacy practice competency domains: gathering information; managing the client and information; following up; and communicating with the client. Knowledge is reported by summary scores of 90 core questions related to cases. Thus, the faculty committee is given valid information to use in judging each student’s level of performance competency and requisite knowledge to provide pharmacy care.

Students receive feedback about exam results in two ways, one informational and the other experiential:

  1. Information in the form of individual summary scores is sent to individual students as a percent of all points available.
  2. The recent experience with a realistic client is the basis for a reflection discussion on what happened and could be improved.

A third way students would receive feedback will be developed in 2009-2010, which is to remediate those who perform the lowest in the class’s first standardized client exam, while there is ample time left on-campus for the shortcomings to be addressed. In this approach, (a) individual goal-setting sessions would be held with the student to tailor goals that need to be worked on; (b) experiential practice assignments would be tailored to maximize the student’s opportunity to work on the goals; (c) the preceptor at experiential practice sites would be asked specifically to create opportunities for the student to practice the goals; and (d) additional academic courses could be negotiated.

A related project undertaken by the School of Pharmacy to ensure that the experience of the students in the program is more unified involves the creation of individual portfolios by students that they maintain throughout their time in the program.

Overall, the standardized client assessment has proven to be a reliable and valid instrument to evaluate the clinical skills we target in the School of Pharmacy. Additionally, students report finding the assessment valuable and worthwhile. Practitioners who participated in case development workgroups also support the project enthusiastically, as evidenced by 18 different pharmacists who contributed six hours each without compensation (seven twice) in the first year of the program.

Portfolio information
The portfolios are living documents that students are required to maintain throughout their time in the pharmacy program. Students enter information about themselves into the document and also reflect on their experiences throughout the program. The goals of requiring the students to complete the portfolios are as follows:

  1. To assist students in identifying and articulating their goals and objectives in the program and their coming careers
  2. To help students gain an appreciation of the depth and breadth of the field of pharmacy and their place in it
  3. To provide students an opportunity to evaluate their strengths and weaknesses in the different areas of pharmacy
  4. To provide a point of dialog for students and advisors when discussing the student’s progress in the program and future career plans
  5. To provide a sense of continuity for the students as they move through the program—creating a project they must follow through with for an extended period of time

Students are first exposed to portfolios during the orientation course they take during the first semester in the pharmacy program. They are required to begin interacting with the document during that class, and in subsequent semesters they continue to fill in information about themselves and their experiences in the program.

The portfolio document (pdf) is accessed electronically and contains space for students to enter information in three main areas: background information, training and educational preparation, and evaluative information. Each of the three main areas has further subdivisions for more specific information.

First, the background information section asks students to articulate their professional philosophy and professional goals, and to document education and work history prior to coming to the school. Next, the training and educational preparation section asks students to describe and reflect on professional activities such as clerkships, elective courses taken, presentations given, and publications. Finally, in the evaluative information section, students are asked to identify what they perceive to be their strengths and weaknesses, to reflect yearly on progress through the program, and to attach formal evaluations and letters of recommendation.

In regard to improving students’ ability to articulate goals and objectives while they are in the pharmacy program, the portfolios have been successful. The process has allowed students to better evaluate their career path and place in the field by requiring reflection about experiences. In addition, the opportunity for self-reflection on strengths and weaknesses has appeared to help students understand what areas they need to improve upon as they progress through the program.

From the standpoint of the faculty who guide the students through interacting with the portfolio, it has provided a window not only into the student’s perceptions of the program, but also into what the clerkship experience is like and what is being done in the field currently. Additionally, the portfolios recently provided information during the school’s accreditation process about student work and attitudes and what the school is doing to foster improved self-awareness in the students.


^Back to top^

Reflections

It is intended that the standardized client program will become a capstone experience within the School of Pharmacy, where students’ progress through the program is dependent on their successful demonstration of professional competency in both the scientific and clinical/counseling domains of practice. For that to occur, the school must continue to develop the model to enhance both the reliability of the assessment, and also to provide a detailed intervention plan for students still in school who do not perform at an acceptable level. If the assessment can be refined to provide additional, detailed information about where students struggle within the different domains of professional competency, then the School of Pharmacy can better prepare its students for their first clerkship and subsequent professional service in a fast-growing and ever-changing field. Besides pharmacy, other professional schools that train practitioners who combine interpersonal service and core discipline knowledge can benefit from this program’s development.

Contact CTE with comments on this portfolio: cte@ku.edu.


^Back to top^

Print Version

Click below for PDFs of all documents linked in this portfolio.

Discipline: 

Portfolio & Poster Search




List of all portfolios & posters

One of 34 U.S. public institutions in the prestigious Association of American Universities
44 nationally ranked graduate programs.
—U.S. News & World Report
Top 50 nationwide for size of library collection.
—ALA
23rd nationwide for service to veterans —"Best for Vets," Military Times
KU Today