Sara E. Monaco, MD

Juan Xing, MD 

 

 

 

In the current era of cytopathology fellow training, determining measures of performance and feedback are important components in order to assess a trainee’s level of performance in the six domains of clinical competency defined by the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS). [1,2]  These core competencies were initially introduced in 1999, and over time became a key part of the newly restructured accreditation of training programs, which is now known as the Next Accreditation System (NAS).[1]  This new system includes milestones that address the core competencies within each specialty, and serve as  competency-based outcomes that can be used as progressive measures of performance prior to unsupervised practice as a physician.

These milestones now guide trainee evaluations, and the NAS requires programs to submit milestone data on the trainees as part of the semiannual evaluation.  Given the importance of milestone evaluations, assessment tools to gauge a fellow’s performance in these domains of the core competencies are critical in order to determine if the fellow can assume progressively more responsibility during their fellowship year, and if they are ultimately competent to progress to independence at the next stage of their career.[2-5]  Evaluation metrics are particularly important in a fellowship program, like cytopathology, where fellows may enter directly from anatomic pathology residency, a combined anatomic and clinical pathology residency, or other fellowships, and thereby enter with different skill sets and abilities.

The current ACGME milestones project was designed to provide an assessment tool for fellows, and serves as a guide to analyze the aspects that are frequently looked at to determine competency within a specialty.[2-5]  Each of the six domains of competency-based milestones has a grade from Level 1 (novice) to Level 5 (expert) and this semiquantitative tool provides a way to analyze a fellow’s level of performance over time.   For cytopathology, the ACGME milestones incorporate a variety of different parameters, including FNA performance, diagnostic knowledge, and honesty/integrity.[2]  The suggested evaluation methods for the milestones include: work-based direct observation, review of reports, self-directed assessment, and retrospective peer review, in addition to other methods.  The implementation of a reliable and accurate evaluation process is essential for competency-based medical education, but not clearly scripted for the individual programs as to the actual ways to assess and measure. [3-6]

Given the need for discrete measures of performance, our program has been using a variety of different internal and external quantitative evaluation parameters to provide a snapshot of the fellow’s performance and provide valuable feedback, which are summarized in Table 1.  The external quantitative measures of performance used include the American Society of Cytopathology (ASC) Progressive Evaluation of Competency (PEC) exam (overall score and scores in individual areas) and the ACGME Case Log System.  The internal quantitative measures of performance have included the cytopathology milestone evaluations that we distribute via our healthcare education management system, MedHub (Minneapolis, MN).  In addition, we have been using our anatomic pathology laboratory information system (AP-LIS; CoPathPlus, Cerner), to create a customized “Discrepant Review Statistics” Cerner report to generate data on our cytopathology fellows, akin to how we generate discrepancy reports on our cytotechnologists.  This allows us to compare the cytopathology fellow diagnosis entered at the time that the fellow reviews the case (e.g. “initial review”), with the final signout diagnosis entered by the attending cytopathologist (“primary review”), and generates a value for the grade of difference between the diagnoses.  The report also calculates the number of reviews for each category and a percent of the total diagnoses that fall in each category, in addition to giving overall summary statistics with the total number of reviews (surrogate for the total number of cases) and the mean grade difference for the diagnoses.(Figure 1)  Discrepancies can then be classified as none (perfect correlation), less than 2-level discrepancy (minor), and greater than or equal to a 2-level discrepancy (major). This was utilized for our cytopathology fellows at approximately the 6th  month & 11th  month of their fellowship year to gauge performance and case volume.  These reports are then shared with the individual fellow and discussed at the clinical competency committee meetings. This internal data allows us to provide a more quantitative and reproducible measure of performance on real-life cases, and provides valuable feedback to the trainee that is not subjective or qualitative from a single evaluator.  Furthermore, we can use our internal data over several years to give them a peer-group comparison, in order to show where their performance falls in relation to other fellows in our program at a similar point in training.  In the future, establishing benchmarks for where a fellow is expected to be at certain parts of their fellowship year may help in providing quantitative measures of progressive responsibility.

Table 1: Examples of Internal and External Quantitative Evaluation Methods for Cytopathology Fellows

Internal Quantitative Evaluation Parameters

·         Average number of cases reviewed (per month, per year, or any discrete time period)

·         Concordance of fellow diagnosis with final signout diagnosis

·         Number or percentage of cases with major diagnostic error (greater than or equal to 2 level discrepancy)

·         MedHub Evaluations using Cytopathology Milestones (graded 1-5 on each milestone) from faculty and fellow (self-evaluation)

External Quantitative Evaluation Parameters

·         ASC PEC Exam- Overall Score (pre, mid, and post results)

·         ASC PEC Exam- Score in individual sections (FNA, Non-GYN, GYN, Lab Operations)

·         ACGME Case Log for Fine Needle Aspirations

 

Figure 1: AP-LIS Discrepant Review Statistics Report for Cytopathology Fellows

Given that the ACGME leaves many of the nuances of how to assess and measure performance up to the discretion of the individual programs, there is a great deal of flexibility and creativity.  However, this also creates a lack of consistency among programs.  The lack of a “one-size-fits-all” approach can be frustrating for some programs, and thus, sharing valuable assessment tools can be valuable within individual specialties with different types of competencies.[5-6]  This study shares some of the quantitative parameters that you can leverage from your LIS in order to help gauge a trainee’s performance on milestones during their cytopathology rotations.  This has been an effective feedback tool for educating the individual trainee, in addition to providing the program director with quantitative measurements of diagnostic performance, other than simply a test score generated from a multiple choice exam, which may more accurately reflect performance in true clinical practice.

*Presented in part at the American Society of Cytopathology (ASC) Annual Scientific Meetings in Phoenix, AZ in November 2017 and Washington, DC in November 2018.

References

  1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system–rationale and benefits. N Engl J Med. 2012 Mar 15;366(11):1051-6.
  1. The Accreditation Council for Graduate Medical Education (ACGME) and The American Board of Pathology, Cytopathology Milestones: https://www.acgme.org/Portals/0/PDFs/Milestones/CytopathologyMilestones.pdf
  1. ACGME Program Requirements for Graduate Medical Education in Cytopathology: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/307_cytopathology_07012017.pdf?ver=2017-05-26-093459-283
  1. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631-7.
  1. Powell DE, Wallschlaeger A. Making sense of the milestones: entrustable professional activities for pathology. Hum Pathol. 2017 Apr;62:8-12.
  1. Sebok-Syer SS, Chahine S, Watling CJ, Goldszmidt M, Cristancho S, Lingard L. Considering the interdependence of clinical performance: implications for assessment and entrustment. Med Educ.2018 Apr 19. doi: 10.1111/medu.13588.
  1. Kinnear B, Warm EJ, Hauer KE. Twelve tips to maximize the value of a clinical competency committee in postgraduatemedical education.  Med Teach. 2018 Jun 26:1-6.

Sara Monaco, MD is Cytopathology Fellowship Program Director and Juan Xing, MD is Associate Program Director at the University of Pittsburgh Medical Center (UPMC).