Review
Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review

https://doi.org/10.1016/j.pec.2015.06.004Get rights and content

Highlights

  • We analyzed 250 assessment forms used during four academic terms.

  • We used the MAAS-Global as the standard for comparison of each individual item on each form.

  • We can measure the ‘true’ caliber of communication skills of OSCE performance.

  • Calibration with only two raters is still a reliable process (G-coefficient = 0.72).

Abstract

Objectives

To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).

Methods

Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.

Results

Papers focusing on generic communication skills, history taking, physician–patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC −0.12–1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.

Conclusions

Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.

Practice implications

Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.

Section snippets

Background

Physicians’ communication skills (CS) have a considerable impact upon quality of health care, whereby good CS improve healthcare outcomes, such as physiologic status, pain control, and emotional health, and significantly increase patient understanding and patient satisfaction [1], [2].

Effective physician–patient communication is essential in ensuring that patients adequately understand their diagnoses, treatment options, medications, plans for referral and prognosis. Dissatisfaction with

Method

A preliminary narrative literature review, pertaining to clinical CS and OSCEs, was conducted by the Principle Investigator (PI), WS, in order to ensure that key points and conceptual frameworks were adequately covered in later search strategies. A list of keywords was developed from the results of this exercise, so that they could form the basis for a more extensive literature search detailed below.

A search was performed in order to identify studies which were published between January 1975

Search results

The initial literature search identified 1998 papers (Fig. 1). After removal of duplicates, 1358 papers remained. By review of the titles and abstracts, 613 were excluded on the basis of irrelevancy. A further 557 papers were excluded as they were not related to OSCEs in undergraduate medical schools. Manual review of the titles and abstracts of the remaining papers identified a further 20 duplicates and 13 non-English language papers, all of which were excluded. In cases where it was not

Discussion

The most striking finding of our study is a demonstrated absence of consensus in rubrics used to assess CS in undergraduate medical education worldwide. Furthermore, it is apparent that there is a clear absence of consensus between researchers in medical education in their interpretation of terminology and in their determination of performance standards in the assessment of CS in different settings. The OSCE is widely utilized to assess CS at undergraduate and postgraduate levels [55]. It is

Conflict of interest statement

All three authors declare no conflict of interest.

Notes on contributors

Winny Setyonugroho, is Lecturer in Health Informatics, Department Health Informatics, Faculty of Medicine & Health Sciences, Universitas Muhammadiyah Yogyakarta, Indonesia and also a PhD student, domain Medical Informatics & Medical Education, Department of Medicine, School of Medicine, College of Medicine, Nursing & Health Sciences National University of Ireland Galway (supervised by Dr Thomas JB Kropmans).

Kieran Kennedy is Lecturer in Clinical Methods and Clinical Practice, Department of

Acknowledgement

The lead author, Winny Setyonugroho, received a PhD scholarship from the Directorate General of Higher Education, Ministry of National Education and Culture of Indonesia.

References (45)

  • M.A. Stewart

    Effective physician–patient communication and health outcomes: a review

    Can. Med. Assoc. J.

    (1995)
  • R.S. Beck et al.

    Physician–patient communication in the primary care office: a systematic review

    J. Am. Board Fam. Pract.

    (2002)
  • W. Levinson

    Physician–patient communication. A key to malpractice prevention

    J. Am. Med. Assoc.

    (1994)
  • P. Maguire et al.

    Key communication skills and how to acquire them

    Br. Med. J.

    (2002)
  • P. Maguire et al.

    Managing the difficult consultation

    Clin. Med.

    (2003)
  • L.M. Noble et al.

    The effect of professional skills training on patient-centredness and confidence in communicating with patients

    Med. Educ.

    (2007)
  • L. Bowleg et al.

    Silences, gestures, and words: nonverbal and verbal communication about HIV/AIDS and condom use in black heterosexual relationships

    Health Commun.

    (2010)
  • M.J. Yedidia et al.

    Effect of communications training on medical student performance

    J. Am. Med. Assoc.

    (2003)
  • G.M. Humphris

    Communication skills knowledge, understanding and OSCE performance in medical trainees: a multivariate prospective study using structural equation modelling

    Med. Educ.

    (2002)
  • R.M. Harden et al.

    Assessment of clinical competence using objective structured examination

    Br. Med. J.

    (1975)
  • J.L. Turner et al.

    Objective structured clinical exams: a critical review

    Fam. Med.

    (2008)
  • B. Hodges et al.

    Evaluating communication skills in the OSCE format: reliability and generalizability

    Med. Educ.

    (1996)
  • Cited by (86)

    View all citing articles on Scopus

    The content was presented at the Association for Medical Education in Europe (AMEE) 2013 conference in Prague, Czech Republic.

    View full text