Assessing validity of standardized patient ratings of medical students’ communication behavior using the Roter interaction analysis system

2.1. Study design and subjects 

We targeted all 121 second year medical students at the Johns Hopkins School of Medicine in the 2003–2004 academic year. Instruction of communication and clinical reasoning occurred during the standard Clinical Skills course which focuses on medical history taking and physical examination. To avoid potential contamination of the Clinical Skills groups, 60 students were randomized to AIME in a three-step process. First, each student was randomized to 1 of 30 Clinical Skills groups. Each group was then randomly assigned to either participate or not participate in AIME. Members of each intervention group were then randomly assigned to 10 different AIME groups so that no members of an AIME group were in the same Clinical Skills group.

2.2. Curricular intervention 

Prior to AIME, there was no formal training in communication and clinical reasoning in the Clinical Skills course. In a previous publication, we describe the details of the AIME curricular development, content and implementation [11]. In brief, AIME is a 6-week course in which small groups of six students and one or two faculty meet on a weekly basis for 3-h sessions. Each session is designed to teach students in a step-wise manner how to: (1) build the doctor–patient relationship, (2) gather an accurate medical history, (3) elicit psychosocial history, cultural and health beliefs, (4) develop a problem list and differential diagnosis, (5) tailor problem solving to individual patient, and (6) elicit patient preferences for decision-making. The primary teaching methods are role play, self-reflection, peer feedback, and group discussions of case scenarios and trigger videotapes.

The role play consists of mock interviews either with another student or a standardized patient. Structured timeouts provide opportunities for feedback on student performance from peers, faculty and self-reflection. Group discussions during timeouts involve a previously described six-step iterative reflective process to elucidate barriers and facilitators to effective communication via an adaptation of the Calgary-Cambridge Observation Guide [12], understand how communication affects the quality of information obtained during history taking, explore challenges to clinical reasoning, and discuss the problem list and differential diagnosis [11]. Role play then resumes with subsequent timeouts using the same format for discussion.

2.3. Communication skills evaluation 

Windish et al. describe in detail the assessment of student performance at baseline and post-intervention [11]. In October 2003, prior to starting the Clinical Skills and AIME courses, all students completed a written test of their communication skills which entailed reviewing a scripted video of a physician interviewing a patient and answering a series of open-ended questions to test their ability to identify select skill sets portrayed in the video. At that time, the students had minimal training in conducting a medical interview. We used this information, in addition to other student characteristics, to demonstrate effective group randomization at baseline. We previously reported that there were no differences in students’ ability to detect selected communication skills [11]. All students then participated in the standard Clinical Skills course. The intervention group simultaneously participated in the 6-week AIME course.

In December 2003, the intervention and control students completed a post-intervention exercise consisting of two standardized patient (SP) encounters during which they conducted a 15-min focused history and physical followed by a 10-min clinical reasoning exercise. Each SP had 8h of training per case, and was observed by the trainers (EGP, DMW and experienced SP trainers at Johns Hopkins) before and during encounters with students to insure accuracy of their presentation and student ratings. The SP cases represented a man with hyperthyroidism and a woman with rheumatoid arthritis. Both cases were designed to have clinical reasoning and communication challenges. Even though neither case was designed to be only a communication skills station, each case had psychosocial issues relating to the effect of the patients’ illnesses on their health, personal relationships and job performance and students were expected to elicit this information. The SPs were trained to respond only to direct questions about these issues.

During the 10-min interval after each encounter, the students completed a clinical reasoning exercise while the SPs rated the students interpersonal and communication skills using a 30-item checklist with a five-point Likert scale (1=poor to 5=excellent). The SPs were previously trained to use this checklist by the standardized patient program directors. In our assessment of student performance, we reduced the checklist to three subscales: data gathering (Cronbach α=0.85), establishing rapport (α=0.90), and patient education and counseling (α=0.86). The subscales were assessed for face and content validity. Given time constraints in the second year medical school curriculum, the baseline written communication skills test was not repeated post-intervention. Moreover, psychometric testing of the instrument was not performed. The evaluation strategy incorporated the usual Clinical Skills course formative evaluation (SP exercise mid-semester).

All SP encounters were videotaped in December 2003. Because funding for this project was limited, and given that Roter et al. have demonstrated significant improvements in residents’ performance with simulated patients pre- and post-communication skills training using only one case per learner [13], we chose to code the communication behaviors used in only one case (hyperthyroidism) to examine concurrent validity of the RIAS with SP ratings of students’ communication skills. We chose the hyperthyroidism case because the case was designed to have more psychosocial issues and communication challenges. Videotapes of the hyperthyroid SP encounter for each student was uploaded to a PC, if available, converted to digitized files and then communication behaviors were coded by trained individuals blinded to the students’ intervention status using the RIAS. The RIAS coders were not study investigators and were trained by faculty and staff employed by the Johns Hopkins Bloomberg School of Public Health.

The RIAS is a valid and reliable instrument developed as a framework for analyzing medical dialogue between physicians and patients in videotaped and/or audiotaped medical encounters [14], [15]. Forty exhaustive and mutually exclusive RIAS communication behavior categories capture complete thoughts (sentence, clause, or single word) expressed by either the patient or the physician referred to as an utterance or unit of talk. These categories group elements of exchange that reflect socioemotional communication (i.e., positive, negative, emotional, partnership building, and social exchanges) and task-focused communication (i.e., asking questions, giving instruction and direction, and giving information). These elements of exchange are then grouped into four primary functions of the medical visit: data gathering, patient education and counseling, responding to patient emotions (rapport), and partnership building.

The RIAS coders rated student and SP global affect using a five-point Likert scale (1=low; 5=high) in the following areas: anger/irritation, anxiety/nervousness, dominance/assertiveness, interest/attentiveness, friendliness/warmth, responsiveness/engagement, sympathetic/empathetic, hurried/rush, and respectfulness.

Each students’ SP encounter was coded by one of two RIAS coders, and 10% of the encounters were double coded by two coders. Intra-coder reliability for rating elements of exchange of student communication skills ranged 0.71–0.98, except for one category social talk (reliability 0.32) in which the frequency of statements was very low.

2.4. Data analysis 

Student characteristics of interest (listed in Table 1) included age (continuous variable), gender (male/female), self-identified race/ethnicity (categorical), college major (science vs. non-science), prior interviewing experience (yes/no) and previous health professional training (yes/no). We compared baseline characteristics using Student's t-test for continuous variables and chi-square analysis for dichotomous or categorical variables. To determine concurrent validity of SP ratings and RIAS coding of students’ communication skills, we examined whether SP ratings detected a difference between AIME and non-AIME students in data gathering, establishing rapport, and patient education/counseling, and whether RIAS coding detected similar group differences in these same skill areas.

Table 1. Baseline student characteristics

	Baseline characteristics	AIME students (N=59)	Non-AIME students (N=61)
	Age (mean)	24.4	24.1
	Female (%)	52.5	52.5
	Caucasian (%)	57.6	54.1
	Non-science college major (%)	19.0	15.0
	Previous interviewing experience (%)	65.5	48.3
	Previous health professional training (%)	32.8*	16.7*

* p=0.04 by chi-square analysis.

Table 2 lists the SP checklist items for each of the three communication subscales. Each communication behavior on the 30-item SP checklist had a five-point Likert scale (1=poor; 5=excellent). Each communication subscale (data gathering, establishing rapport, patient education/counseling) had five items. Therefore, possible mean scores for each subscale ranged 5–25. Each average score was then converted to a five-point score based on the number of items in each subscale. This five-point score corresponds to the SP checklist Likert scale which we used to interpret SP ratings of each subscale. We compared SP ratings (continuous variable) of AIME and non-AIME students’ data gathering, establishing rapport and patient education/counseling for each SP case separately. We used Student's t-test to compare differences in average SP ratings between AIME and non-AIME students, and a multivariable analysis to determine the independent influence of baseline characteristics (age, gender, college major, prior interviewing experience, prior health professional training) on each outcome. We then compared SP ratings of each groups’ performance on these subscales for both cases combined by averaging SP ratings across the two cases for each subscale. For these combined scores, we conducted the aforementioned analysis.

Table 2. Communication behaviors measured by the standardized patient checklist

	Data gathering
	Elicits information in an organized manner
	Asks clear and unambiguous questions
	Uses vocabulary at level of patient understanding
	Listens carefully without interruptions allowing sufficient time for a response
	Uses restatement, reflection and clarification to verify information and indicate active listening
	Rapport
	Greets patient warmly
	Demonstrates courteous and professional behavior
	Conveys a sensitive and caring attitude
	Demonstrates interest in the patient
	Displays an open, receptive and interested posture
	Patient education and counseling
	Uses clear, organized explanations
	Provides enough information to answer questions
	Encourages questions
	Discusses options and helps patient decide what to do
	Encourages patient to give opinions about treatment plans

For RIAS coding, data gathering utterances/statements were defined as closed and open questions on medical, therapeutic, lifestyle, psychosocial or other issues. Establishing rapport was defined as encompassing several types of behaviors including laughter, approval, compliments, shows agreement or understanding, disagreements, direct/general criticism, empathy, concern/worry, reassures, partnership, self-disclosure and personal remarks. Patient education and counseling was defined as information giving on medical, therapeutic, lifestyle, psychosocial or other issues. To determine the students’ RIAS communication score for the hyperthyroid case, we calculated the total number of utterances for each functional category (data gathering, rapport, patient education/counseling). We used Student's t-test to compare differences in average scores between AIME and non-AIME students, and a multivariable analysis to determine the independent influence of baseline characteristics (age, gender, college major, prior interviewing experience, prior health professional training) on each outcome. All analysis was performed using Stata Statistical Software 8.0 Intercooled (Stata Corporation, College Station, TX 2002).

3.1. Student characteristics 

Among the 121 second year medical students, 120 completed the standardized patient exercise. AIME and non-AIME students were similar in age, gender, self-identified race/ethnicity, college major and previous interviewing experience; however, a higher proportion of AIME students reported previous health professional training (32.8% vs. 16.7%, p=0.04) (Table 1).

3.2. Standardized patient ratings of students’ communication skills 

Table 2 lists the SP checklist items included in the communication subscales. There were no statistically significant differences in SP ratings of AIME compared to non-AIME students’ data gathering, rapport building and patient and education and counseling skills when each SP case was analyzed separately (Table 3). However, when comparing average student performance for both SP cases combined, AIME students’ rapport-building skills were rated higher (mean [S.E.]: 4.09 [0.15] vs. 3.91 [0.15], p=0.05) after adjusting for prior health professional training. There were no significant differences in ratings of data gathering and patient education and counseling subscales. Moreover, review of SP ratings revealed that the average score for each subscale was 3 or higher (3=good to 5=excellent) for each case individually and for combined scores for both cases.

Table 3. Standardized patient ratings of 59 AIME and 61 non-AIME students’ communication behaviorsa

		Standardized patient rating hyperthyroidism			Standardized patient rating rheumatoid arthritis			Standardized patient rating both cases
		Mean (S.E.)		p-Valueb	Mean (S.E.)		p-Valueb	Mean (S.E.)		p-Valueb
		AIME	Non-AIME		AIME	Non-AIME		AIME	Non-AIME
	Data gathering	3.62 (0.08)	3.50 (0.08)	0.29	3.84 (0.10)	3.81 (0.10)	0.83	3.66 (0.15)	3.58 (0.15)	0.33
	Rapport	3.94 (0.10)	3.77 (0.10)	0.22	4.29 (0.10)	4.13 (0.10)	0.16	4.09 (0.15)	3.91 (0.15)	0.05
	Patient education and counseling	2.85 (0.10)	2.77 (0.10)	0.49	3.27 (0.14)	3.24 (0.13)	0.87	3.00 (0.20)	3.03 (0.20)	0.62

a SP rating scale: 1, poor; 2, adequate; 3, good; 4, very good; 5, excellent. b All analyses are adjusted for students’ previous health professional training.

3.3. Frequencies of student communication behaviors obtained from RIAS coding of videotaped SP encounter 

Among the 120 students with videotaped SP encounters, 10 videotapes were inaudible (4 AIME; 6 non-AIME), and 13 tapes were missing (4 AIME; 9 non-AIME). Ninety-seven students (51 AIME; 46 non-AIME) had videotapes available for RIAS coding to examine concurrent validity of SP ratings of communication behaviors.

Table 4 lists the RIAS categories of communication behaviors. The RIAS showed that AIME students used a higher mean frequency of rapport-building statements (mean [S.E.]: 60.4 [2.7] vs. 52.1 [2.8], p=0.03) after adjusting for prior health professional training, especially statements of partnership and reassurance/optimism (1.4 [0.2] vs. 0.6 [0.2] and 6.4 [0.7] vs. 4.0 [0.7], p=0.02, respectively). There were no differences in frequencies of data gathering, patient education/counseling utterances by student AIME-status, age, gender, ethnicity or college training after adjusting for prior health professional training. Specifically, there were no significant differences between groups in the frequency of biomedical and psychosocial/lifestyle related statements.

Table 4. Frequencies of students’ communication behaviors during one SP encounter obtained from RIAS coded videotapesa

		Mean (S.E.)		p-Valueb
		AIME (N=51)	Non-AIME (N=46)
	Data gathering	44.7 (2.5)	41.7 (2.3)	0.39
	Closed question—medical	13.6 (1.2)	14.9 (1.2)	0.44
	Closed question—therapeutic	0.1 (0.1)	0.3 (0.1)	0.12
	Closed question—lifestyle	3.5 (0.6)	3.1 (0.6)	0.69
	Closed question—psychosocial	0.6 (0.1)	0.4 (0.1)	0.12
	Closed question—other	0.5 (0.1)	0.4 (0.1)	0.31
	Open question—medical	19.0 (1.0)	17.3 (1.0)	0.23
	Open question—therapeutic	1.5 (0.2)	1.1 (0.2)	0.10
	Open question—lifestyle	2.9 (0.4)	1.9 (0.4)	0.09
	Open question—psychosocial	2.8 (0.4)	2.0 (0.4)	0.13
	Open question—other	0.1 (0.1)	0.2 (0.1)	0.09
	Bid for repetition	0.1 (0.1)	0.2 (0.1)	0.42
	Rapport building	60.4 (2.7)	52.1 (2.8)	0.03
	Empathy	0.8 (0.1)	0.5 (0.2)	0.15
	Legitimation	1.20 (0.2)	0.7 (0.2)	0.13
	Concern/worry	3.4 (0.4)	2.3 (0.4)	0.07
	Reassures/optimism	6.4 (0.7)	4.0 (0.7)	0.02
	Partnership	1.4 (0.2)	0.6 (0.2)	0.02
	Self-disclosure	0.9 (0.2)	0.7 (0.2)	0.46
	Laughs	3.5 (0.4)	2.8 (0.4)	0.28
	Approves	6.3 (0.6)	4.4 (0.6)	0.02
	Compliments	0.02 (0)	0 (0)	0.36
	Agreements	29.3 (1.8)	30.1 (1.9)	0.77
	Direct criticism/disagreement	0.1 (0.6)	0.02 (0.6)	0.15
	General criticism/disagreement	0 (0)	0 (0)	–
	Personal remarks	6.0 (0.4)	5.3 (0.4)	0.19
	Patient education and counseling	19.0 (2.3)	16.8 (2.4)	0.53
	Gives information—medical	9.0 (1.6)	9.0 (1.7)	1.0
	Gives information—therapeutic	7.3 (1.0)	5.4 (1.0)	0.17
	Gives information—lifestyle	0.1 (0.6)	0.1 (0.1)	0.71
	Gives information—psychosocial	0.3 (0.1)	0.4 (0.1)	0.30
	Gives information—other	1.4 (0.3)	1.3 (0.3)	0.81
	Counsels—medical/therapeutic	0.5 (0.2)	0.5 (0.2)	0.94
	Counsels—lifestyle/psychosocial	0.4 (0.2)	0.1 (0.2)	0.19

a Data presented in the table represents RIAS coding of the hyperthyroidism case. b All analyses are adjusted for students’ previous health professional training.

Finally, there were no significant differences in the RIAS ratings of AIME students’ compared to non-AIME students’ global affect with respect to anger/irritation, anxiety/nervousness, dominance/assertiveness, interest/attentiveness, friendliness/warmth, responsiveness/engagement, sympathetic/empathetic, hurried/rush, and respectfulness (data not shown).

4.1. Discussion 

To our knowledge, this is the first study to examine the concurrent validity of SP ratings of medical students’ communication behaviors with a well established, previously validated coding instrument such as the RIAS. We designed the AIME curriculum to teach second year medical students strategies for integrating patient-centered communication and clinical reasoning while understanding the connection between biomedical and psychosocial aspects of patient care. In a two-station standardized patient exercise, SPs rated AIME students’ rapport-building skills higher than control students. An independent assessment of the students’ communication skills, accomplished by RIAS coding of one SP case confirmed that AIME students used more rapport-building statements that involved more statements of reassurance/optimism and partnership. There was no difference between AIME and non-AIME students in global affect during the interviews. Also, AIME and non-AIME students’ data gathering, patient education and counseling skills did not differ significantly. AIME did not achieve any measurable differences in the balance of psychosocial and biomedical communication behaviors used by students.

We expected our AIME students to display more patient-centered communication behaviors in their encounters with SPs than their non-AIME counterparts. Prior to the SP exercise, AIME students had opportunities to receive structured feedback on their communication skills from SPs, student peers, faculty and to engage in self-reflection using a communication skills observation guide [11]. The guide helped us to systematically train students to conduct all three functions of the medical interview (data gathering, rapport building, patient education and counseling). Each small group session consistently taught AIME students how to balance use of open- and closed-ended questions, avoid medical jargon, respond to verbal/non-verbal cues and patient emotions, display empathy and active listening, and elicit patient beliefs and preferences. Non-AIME students had no exposure to a structured curriculum in these aspects of medical interviewing. Results from the RIAS indicate that AIME students may have focused more on reassuring the patient, validating patient concerns and displaying partnership behaviors.

This study confirms that SP ratings of students’ communication behaviors are a valid approach to curricular evaluation. While many studies regarding the reliability and validity of SP ratings examine the objective structured clinical examination (OSCE) as a measure of individual learner clinical competence [16], few studies examine non-OSCE type SP exercises to assess group performance after communication skills training. We used a two-station SP exercise which might arguably be an insufficient number of cases to draw conclusions about students’ performance in any given domain of communication behaviors (e.g., data gathering/establishing rapport/patient education and counseling). Van der Vleuten et al. suggest that approximately 12 SP encounters may be required to achieve generalizability to support inferences about learner performance [17]. It is not clear, however, whether this threat to validity applies to assessments of group performance. To minimize threats to the validity of our study, SPs underwent standardized training to insure accurate case portrayal as well as appropriate use of the communication skills rating scale. Moreover, standardized patient monitoring during the performance examination allowed timely correction of any slippage from standard case portrayal. Such measures reduce systematic errors such as misrepresentation of the construct of interest which may lead to variance in student performance and SP ratings [18], [19].

While SP ratings are commonly used for formative or summative evaluations of learner communication skills [20], prior studies suggest that SP ratings tend to favor learners [21], [22]. Our study confirmed this tendency as most students’ communication scores, regardless of AIME-status, were 3 or higher on a five-point Likert scale, leading us to question the validity of SP ratings. However, the RIAS coders, independent raters blinded to the students’ curricular assignment, confirmed differences between AIME and non-AIME students’ rapport-building skills.

Notably, the SP checklist items and RIAS communication categories are not equivalent; yet both performance assessments yielded similar findings. The RIAS provided descriptions of specific verbal communication behaviors used and global affect displayed during the student–SP interaction, but it did not indicate whether the behaviors were used appropriately. The SP checklist, in contrast, required the SPs to incorporate judgments as to whether verbal and non-verbal communication behaviors were used appropriately during the interview (e.g., responsive to the context of the interaction). Stiles argues that while descriptive measures of interviews reveal insights into the process of interviewing, they do not necessarily distinguish which interviews are good [23]. “Skillful interviewing requires responsiveness to the emerging circumstances rather than simply increasing or decreasing the use of particular components” [23]. Both SP cases (hyperthyroidism and rheumatoid arthritis) were embedded with psychosocial issues relating to the effect of the patients’ illnesses on their health, personal relationships and job performance. SPs were trained to convey certain emotions as these issues were explored by the interviewer. The SPs’ global affect did not differ between interviews with AIME and control students. Therefore, AIME students may have elicited the psychosocial context of the patients’ illness in a manner that was more responsive to the emotional tone set by the SPs compared to non-AIME students. Indeed, the RIAS revealed that AIME students used more statements that have been considered emotional in nature (e.g., empathy, legitimization, reassurances, optimism, concerns, self-disclosures) [14].

Our study findings must be interpreted in light of several limitations. First, we did not have pre-intervention and post-intervention SP encounters to assess changes in SP ratings and RIAS coding of students’ communication behaviors. Although we can conclude that AIME students’ rapport-building skills differ from control students, we can neither conclude definitively that this difference is only due to AIME nor can we determine the effect size of any changes in performance attributable to AIME. Second, we were unable to analyze videotapes for 13.5% (n=8) of AIME students and 24.6% (n=15) of non-AIME students. Our study findings, therefore, could be biased in favor of AIME students. However, the aforementioned Roter study involved only 28 participants and was still able to detect group differences in communication behaviors [13]. Third, even though we used a two-station SP exercise to evaluate communication skills, we used RIAS to analyze only one of the SP cases and we used the same case for each student. Prior studies have suggested that the quality or nature of learners’ communication skills may be case specific [24]. In other words, the presenting problem may serve as a source of variation in a learner's communication behaviors. One could argue that multiple cases should have been used to assess individual communication behaviors via the SP exercise and the RIAS coding. However, we did not use the SP encounters to assess individual skills. Instead, we sought to evaluate curricular effectiveness via assessing differences in group performance based on intervention status. Given that two SP encounters were recorded for all students, the study findings may have been strengthened if we found the same results after RIAS coding of both SP cases for all students or randomly selecting one of the two cases for each student. Limited funding for this study precluded the former option. Finally, we did not perform Bonferroni adjustments for the analysis of the RIAS variables. Although we performed 33 individual regression analyses, the results we obtained are ones that we hypothesized and that have theoretical meaning. Our focus on rapport-building behaviors was guided by our previous study that used SP ratings to evaluate the success of AIME. Because of the consistency of the association across several items in the same domain of RIAS and SP ratings, it is unlikely that the findings related to rapport building and AIME assignment occurred by chance alone. Limitations of Bonferroni adjustments have been previously described [25]; therefore, we opted against using a Bonferroni adjustment in this study.

4.2. Conclusion 

This study contributes new information to the literature regarding the validity of SP ratings of students’ communication behaviors. While our review of the literature found studies that compare SP ratings to faculty observer ratings [22], [26], we could not identify studies similar to ours which assess concurrent validity of SP ratings with audiotape or videotape analysis of learner communication behaviors. Although the specific communication behaviors measured by the SP rating scales differed from that of the RIAS, both methods detected a difference in students’ rapport-building skills representing some degree of concurrent validity.

4.3. Practice implications 

Future studies in medical education should further examine the minimum number of SP ratings needed to effectively evaluate communication skills curricula when resources are limited. Having valid, cost-saving tools with which to measure curricular effectiveness is important when medical education accreditation organizations are increasingly requiring institutions to raise standards of training.