Daniel Schumacher

Chapter 1

Chapter 1 describes recent calls to link educational outcomes and patient care outcomes in graduate medical education. Achieving this goal is the foundation of competency-based medical education, but current gaps in quality improvement training and resident performance assessment practices hinder efforts to achieve it. Concerning gaps in assessment, there is both unwanted variation in assessments as well as failure to align assessment with patient care. In this chapter, we propose that patient-centered performance assessment, specifically entrustable professional activities (EPAs) and quality measures that are relevant to the work that residents complete (which we term resident-sensitive quality measures, or RSQMs), is a way to overcome these barriers. By closing these gaps, we will begin to provide the ability to better link educational outcomes with patient care outcomes. However, EPAs currently lack sufficient empirical evidence despite their international popularity over the past decade, and there are also insufficient quality measures for the work that residents complete. This situation leads to the main research questions for this dissertation:

What are the characteristics of developing and implementing quality measures, which we term resident-sensitive quality measures (RSQMs), that are largely attributable to residents and important to patient care?

What is the association between entrustment decisions made about residents and their performance on RSQMs?

The studies in this dissertation took place in the U.S. pediatric residency context. Specifically, we focused in the pediatric emergency department for our initial efforts in this novel research direction.

Chapter 2

Chapter 2 extends the call to focus on new approaches of performance assessment to better link educational and patient care outcomes, arguing that the future of high-quality care depends on better physician performance assessment. This chapter proposes a three-pronged research agenda in this area. First, EPAs should be implemented and studied broadly. Second, the association between entrustment decisions made about residents and quality measures for the care they provide should be investigated. Finally, with only 2% of global health expenditure dedicated to health education for all professions combined, these efforts should receive funding and resource prioritization from public and private sectors.

Chapter 3

The research program presented in this dissertation focuses on RSQM performance and entrustment decisions for individuals. However, given the recent focus on health care provided by teams, it is important to justify the focus on individuals. Chapter 3 argues the importance of focusing on individuals (by considering attribution of care) as well as teams, programs, and systems (by considering contribution to care). Contribution focuses on the various degrees different entities contribute to an outcome, and it plays an important role in analyzing program and system-level outcomes that inform program evaluation and program-level improvements for the future. Equally important in health care are the role of the individual and aspects of care that can be attributed to an individual team member. Providing individuals with data related to their performance on the team has value in driving their personal improvement. Furthermore, we graduate, certify, and credential individual physicians, not teams, for medical practice. Additionally, incentive-based payment models focus on the quality of care provided by individuals, making it important to be able to determine outcomes of performance attributed to individuals. In this chapter, we explore how attribution and contribution analyses can be used in a complimentary fashion to discern what outcomes can and should be attributed to individuals as well as to teams and programs.

Chapter 4

Chapter 4 describes efforts to begin closing the training and assessment gap with respect to quality measures available for use with residents through the development of RSQMs for use in the pediatric emergency department setting. In this study, we used a nominal group technique (NGT) and Delphi process with faculty and fellow supervisors at Cincinnati Children’s Hospital Medical Center to develop (through the NGT) and prioritize (through the Delphi) possible RSQMs for acute asthma exacerbation, bronchiolitis, and closed head injury encounters. Groups were asked to generate (in the NGT) and rate (in the Delphi) possible RSQMs based on two criteria: 1) importance of the measure to the illness of interest and 2) likelihood that a resident, and not another member of the team or the team collectively, completes the measure. The NGT produced 67 measures for asthma, 46 for bronchiolitis, and 48 for closed head injury. These were used to populate the first round of the Delphi process. After two rounds, 18 measures for asthma, 21 for bronchiolitis, and 21 for closed head injury met automatic inclusion criteria. In round three, participants categorized the potential final measures by their top 10 and next 5. Prioritized measures often fell into one of three categories: 1) appropriate medication use, 2) documentation, and 3) information provided at patient discharge. In summary, this study provides a template for identifying and developing RSQMs that may help to promote high-quality care delivery during and following training.

Chapter 5

Chapter 5 extends the work of chapter 4 in two ways: 1) replicating the process of developing RSQMs for asthma, bronchiolitis, and closed head injury in the pediatric emergency department setting with pediatric residents, and 2) engaging resident and supervisor stakeholders to develop and inform next steps in creating RSQMs. Like faculty and fellow supervisors, residents in the NGT and Delphi groups placed considerable focus across the three illnesses of interest on measures that pertained to 1) appropriate medication use, 2) documentation, and 3) information provided at patient discharge. Subsequent focus groups with residents as well as faculty and fellow supervisors highlighted hospital medicine and general pediatrics as priority areas for developing future RSQMs but also noted contextual variables that influence the application of similar measures in different settings (e.g., team structure of general pediatrics clinic may limit the utility of some measures). Residents and supervisors had both similar and unique insights into developing RSQMs. For example, the resident NGT produced more measures than the supervisor NGT, and both focus groups felt this may be because residents work on the front line and know more details of care provided there. However, supervisors also noted that residents may not be able to prioritize well or do not know resources available to help complete their work, and residents noted that supervisors may not know details that go into actually providing frontline care. In summary, this chapter builds on the work of chapter 4 to continue to pave the path forward in developing future RSQMs by exploring specific settings, measures, and stakeholders to consider when undertaking this work.

Chapter 6

Chapter 6 presents the range of individual and collective performance for the RSQMs introduced in chapters 4 and 5 when these RSQMs are used to assess pediatric residents working in the pediatric emergency department. During the 2017–2018 academic year, pediatric residents in the Cincinnati Children’s Hospital Medical Center pediatric emergency department were assessed using RSQMs for patient encounters focused on acute asthma exacerbation (21 RSQMs), bronchiolitis (23), and closed head injury (19). Of included encounters, 83 residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with closed head injury. Residents had the opportunity to meet most RSQMs, but exceptions existed (e.g., patients with asthma exacerbation presenting to the medical resuscitation area had initial orders placed by the supervising physician, per protocol, precluding residents from meeting some RSQMs that pertained to initial orders). In encounters presenting the opportunity, there was a wide range of frequency for how often residents performed RSQMs. One closed head injury measure was met in all encounters. Across illnesses, another 9 RSQMs were met in almost all encounters, but most RSQMs were not met, with a frequency ranging from several times to the majority of encounters. To determine the proportion of individual RSQMs correctly performed out of total possible RSQMs that could have been performed, we generated a composite score for each encounter. RSQM composite scores demonstrated notable range and variation: asthma mean, 0.81 (SD: 0.11); bronchiolitis mean, 0.62 (SD: 0.12); closed head injury mean, 0.63 (SD: 0.10). In summary, chapter 6 demonstrates that individual and composite RSQMs capture and discern resident performance across patient encounters, opening the door for their use in other contexts and by other specialties.

Chapter 7

Chapter 7 explores the association between RSQM composite scores for individual pediatric emergency department encounters, supervisor entrustment decisions (as measured by supervision level assigned) for those encounters, and patient acuity and complexity for those encounters. The 83 residents and 299 patient encounters for asthma, bronchiolitis, and closed head injury presented in chapter 6 were used for the study presented in this chapter. To measure the association of RSQM composite scores with the other variables of interest, we used mixed models to account for observations nested within residents. Entrustment decisions were positively associated with asthma RSQM composite scores (beta-coefficient 0.03; P = .0004), but there was no significant association between RSQM composite scores and entrustment decisions for bronchiolitis or closed head injury. While statistically significant, the positive linear relationship for asthma is likely not educationally or clinically significant because RSQM composite scores only increased 0.03 (out of 1) for every 1 (out of 7) point increase in entrustment score. A positive relationship would be expected between entrustment decisions and RSQM composite scores. Given that RSQMs are a more objective measure of performance, the findings of this study should prompt consideration of whether RSQM performance data should help inform entrustment decisions. RSQM composite scores were significantly higher when acuity was also higher and significantly lower when acuity was also lower for both asthma (P = .0009) and bronchiolitis (P = .01). However, RSQM composite scores were almost identical for the different levels of acuity for closed head injury (P = .94). There was no association between RSQM composite scores and patient complexity.

Chapter 8

Having established initial implementation evidence for using RSQMs in the pediatric emergency department, chapter 8 sought to consider how RSQM data might be used by those who make assessment decisions outside the pediatric emergency department. The study presented in this chapter achieved this goal by exploring how individual clinical competency committee (CCC) members interpret, use, and prioritize RSQM data inserted into their usual review processes. In this constructivist grounded theory study, we purposively and theoretically sampled 19 pediatric residency CCC members from Cincinnati Children’s Hospital Medical Center. Participants were provided a resident assessment portfolio consisting of performance ratings and narrative comments for five rotations as well as RSQM data for one of these rotations. They were asked to review with the purpose of making a decision about the residents’ ability to care for patients presenting with common, acute problems (a general pediatric entrustable professional activity). Data collection consisted of two phases: 1) observation and think aloud in which participants reviewed the resident portfolio, and 2) semistructured interviews to probe participant reviews. Analysis moved from close reading to coding and theme development followed by the development of a model illustrating how themes interact with one another. We identified five dimensions for how participants view and use RSQMs: 1) ability to orient to RSQMs: confusing to self-explanatory; 2) propensity to use RSQMs: reluctant to enthusiastic; 3) RSQM interpretation: requires contextualization to self-evident; 4) RSQMs for assessment decisions: not sticky to sticky; and 5) expectations for residents: potentially unfair to fair use of RSQMs. These dimensions interact with one another to generate three profiles regarding how participants used RSQM data: eager incorporation, willing incorporation, and disinclined incorporation. The former two profiles were most common, with most participants using RSQMs to some extent, supporting the inclusion of RSQMs as resident assessment data for CCC review.

Chapter 9

As RSQMs continue to be developed, it will be important to optimize ways to extract them from the electronic health record (EHR) and to automatically identify which residents they should be attributed to in settings where multiple residents provide care to a given patient, such as the inpatient environment. Chapter 9 sought a method for attributing care of individual patients to individual interns based on “footprints” in the EHR. Primary interns caring for patients on an internal medicine inpatient service were recorded daily by five attending physicians of record at University of Cincinnati Medical Center in August 2017 and January 2018. These records were considered gold standard identification of primary interns. The following EHR variables were explored to determine representation of primary intern involvement in care: postgraduate year, progress note author, discharge summary author, physician order placement, and logging clicks in the patient record. These variables were turned into categorical attributes (e.g., progress note author: yes/no), and informative attributes were selected and modeled using a decision tree algorithm. A total of 1,511 access records were generated; 116 were marked as having a primary intern assigned. All variables except discharge summary author displayed at least some level of importance in the models. The best model achieved 78.95% sensitivity, 97.61% specificity, and an area under the receiver-operator curve of approximately 91%. Thus, this study successfully predicted primary interns caring for patients on inpatient teams using EHR data with excellent model performance. These results provide a foundation for attributing patients to primary interns for the purposes of determining profiles of patient diagnoses by trainee, supporting continuous quality improvement efforts in graduate medical education, and ultimately attributing RSQMs to residents.

Chapter 10

Chapter 10 details calls for ensuring high value health care and posits that medical education can and should drive progress toward this goal. We argue that getting medical education to the point where it can do this work will require a clear, sustained focus on the patient, which is not traditionally a notable focus of medical education research or practice. Patient-focused assessment efforts, such as RSQMs, can help achieve this goal. The summary of our research is that RSQMs developed through consensus group methodology largely do reflect the performance of residents and that performance varies widely across residents and encounters. Future development of RSQMs will need to consider engaging other stakeholders and expanding to other settings and specialties. Our research suggests future settings for pediatrics, including hospital medicine and primary care outpatient pediatrics, but these efforts should also be undertaken in other medical specialties as well as surgical and hospital-based specialties. Completing this work should include engaging with stakeholders we have not yet involved in our work to date, including nurses and patients/families.

Our findings for the association between RSQMs and entrustment decisions made about residents varied. RSQM composite scores were significantly higher for residents entrusted with unsupervised practice or the ability to supervise others than for residents entrusted to lower supervision levels for all conditions (asthma, closed head injury, and all conditions combined) other than bronchiolitis. This finding offers validity evidence for RSQM composite scores; however, the absolute difference was modest. Additionally, when considering each supervision level and not combining them into two categories (at least entrusted with unsupervised practice or not entrusted to that level), a significant positive linear relationship between RSQM composite scores and entrustment decisions was only seen for asthma. However, that observation for asthma is likely not educationally or clinically significant. We believe this finding may be more problematic for the consideration of entrustment as an assessment framework than for using RSQMs to inform resident assessment. Specifically, we posit that RSQMs may help to improve the veracity of entrustment decisions.

Despite the need for further understanding and validity evidence, we believe patient-focused assessment efforts, such as RSQMs and entrustment, should be embraced given the promise to ensure high value care embodied in their attention to the patient. However, hazards to their use exist. There can be reluctance to adopt new ways of approaching assessment. Nonetheless, recent conversations in medical education about moving away from psychometrics and toward welcoming the subjective in performance assessment suggest change is possible. With calls for ensuring high value health care as well as uniting the missions of medical education and health care delivery, medical education may be at a crossroads. Efforts such as RSQMs can help the community take a clear step toward focusing on the patient in our assessment efforts.