Laury De Jonge

Competency-based medical education (CBME) focuses on letting learners integrate and apply their knowledge and skills in real-life practice. This approach calls for ongoing assessment of authentic clinical tasks to foster learning. For this reason, workplace-based assessment (WBA) has become an important cornerstone of CBME. In our introductory chapter (Chapter 1), we explain that WBA fulfils both a formative and summative function: to drive learning and monitor the trainee’s competence development (formative) and to enable decisions on the trainee’s achievement during training (summative). The authenticity of the clinical environment implies that WBA is complex and typically influenced by uncontrolled variables such as case difficulty, patient mix and patient numbers. Moreover, previous research has clearly demonstrated that the quality of WBA mainly depends on how stakeholders (e.g., clinical supervisors, trainees, programme directors) use the assessments rather than on the intrinsic qualities of the instruments and methods used to evaluate clinical performance. Performance interpretations in the clinical workplace and the resulting judgements, feedback and decisions are the product of a continuous interplay between a broad range of cognitive and social factors. From a socio-cultural stance, performance assessment can be seen as a ‘socially situated interpretative act’; it is always conceptualised and constructed according to the conceptions (i.e., conscious or subconscious beliefs, concepts, meanings, experiences and mental images) of its individual stakeholders who are situated in and affected by a unique social context. Social interactions in WBA are not limited to the clinical workplace itself (e.g., interactions within a supervisor-trainee dyad or with a patient), but potentially extend to stakeholders at organisational (e.g., facility managers, supervisor colleagues), institutional (e.g., education programme designers) and national level (e.g., accreditation bodies, governmental medical legislation). This means that social interactions at various levels influence stakeholders’ conceptions of WBA and their associated behaviours.

By bringing stakeholders’ conceptions to the surface and rethinking their influence on WBA strategies, this dissertation aims to contribute to a meaningful understanding of WBA practices. The main objective of the research described in this dissertation is to further our understanding of stakeholder conceptions of WBA and their impact on the assessment strategies used during postgraduate medical training.

In Chapter 2, we studied various stakeholders’ conceptions of WBA by exploring the perspectives and beliefs that underlie their approaches to WBA in a postgraduate medical training programme. We systematically investigated these conceptions using Q methodology that combines aspects of qualitative and quantitative methodological approaches. To this end, we purposively sampled stakeholders (i.e., supervisors, trainees, institutional teachers, programme directors) from two general practice training institutes between which they were equally distributed, asking them to perform individual Q sorts by ranking 48 statements about WBA. To uncover patterns in the ranking of statements, we performed a by-person factor analysis, after which we iteratively interpreted and described each factor. We extracted five different factors or ‘perspectives’ on performance assessment: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflected both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the perspectives of the various stakeholders, we found that these could be positioned on two continua (axes), specifically ‘the locus of regulation of learning’ (i.e., self-regulated versus externally regulated learning) and ‘the extent to which assessment should be standardised’ (i.e., tailored versus standardised assessment). The first continuum concerned the importance attributed to self-regulation in learning and assessment. Participants’ views on this theme varied widely: from believing that trainees should actively self-direct their learning, take responsibility and show initiative in the assessment process, through feeling that the responsibility to identify learning needs by means of performance evaluations and feedback rests with both the supervisor and the trainee, to the perspective that supervisors should actively direct trainees’ learning and take the lead and responsibility for the assessment process. The second continuum where stakeholders’ conceptions of WBA diverged concerned the extent to which they felt both WBA instruments and the assessment process itself should be standardised. Stakeholders’ preferences with regard to the level of standardisation ranged from total reliance on idiosyncratic expert judgement based on contextualised task requirements and assessment criteria tailored to trainees’ individual learning needs, through the tailoring of assessment criteria to the clinical task and context, to objectifiable and standardised approaches to WBA, using predefined assessment tasks and detailed assessment criteria. Differing perspectives may variously affect stakeholders’ acceptance, use and, consequently, the effectiveness of assessment programmes. The central practical implication of this study is therefore that continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model on WBA conceptions and practices.

In the following chapters of this dissertation, we studied the influence of stakeholder conceptions on two key strategies in WBA, namely entrustment decision-making and direct observation of clinical performance.

Chapter 3 reports a study that aimed to gain insight into the process of supervisors’ decision-making strategies underlying the entrustment of clinical tasks at the start of one-to-one supervisor-trainee working relationships. For that purpose, we gathered naturalistic data using non-participant observations that were triangulated with semi-structured interviews with supervisors in four general practice settings. Data were analysed using a constructivist grounded theory (CGT) approach. Consistent with the principles of CGT, we collected and analysed data in iterative cycles, and used memo writing in order to produce an audit trail from data to interpretation. Our study demonstrated that, in the earliest phase of training, supervisors based their entrustment decisions more on generic trainee qualities (i.e., trainees’ self-reflexivity, knowing their limitations and asking for help in time) than on perceived clinical competence. In order to make these decisions, supervisors’ primary concern was to create a safe working and learning environment in which trainees could and would ask for timely supervision. Supervisors used individual strategies to offer trainees a challenging learning climate, while at the same time minimising the risks for patients and themselves. These strategies were influenced by interrelated factors, such as supervisors’ propensity to trust, their previous experiences with trainees, and their perspective on learning and teaching. We found that positive past experiences made supervisors entrust tasks more easily to their current trainee, while negative experiences could lead to a more controlling approach. Moreover, we found that supervisors’ propensity to trust affected their trustworthiness perceptions and resulting entrustment strategies. This finding is particularly relevant when little information about an individual’s trustworthiness is available, such as at the start of a supervisory relationship. Especially at the start of medical residency training, supervisors’ propensity to trust and their resulting supervisory style may thus impact on the extent to which trainees are allowed to control their own learning and supervision processes. Supervisors must therefore realise that their individual experiences, assumptions and beliefs influence their approaches to entrustment and, consequently, trainees’ learning opportunities. To ensure that entrustment strategies are well aligned with the needs of the stakeholders involved, it seems especially important that supervisor and trainee regularly discuss and explicate mutual expectations and responsibilities regarding the supervision initiative.

In Chapters 4 and 5, we examined the influence of stakeholder conceptions on the use of observation during WBA. Performance observations play a pivotal role in CBME, including postgraduate training. They are an effective strategy to monitor trainees’ day-to-day performance and provide supervisors with important information that can be used to give feedback on a variety of competencies. Obtaining information via observations is a prerequisite for robust decision-making on a trainee’s competence development. Moreover, performance observations enable decisions about levels of entrustment in task performance, thereby maximising the potential for trainee learning. Incorporating observations in medical training programmes is therefore strongly recommended. Yet, despite its undisputed importance to credible feedback and assessment, observations are not implemented in WBA as well as they should.

Previous research has demonstrated that supervisors may not only experience practical barriers (e.g., time constraints, busy practice) and socio-cultural barriers (e.g., observations interfering with trainees’ quest for autonomy) to direct observations in healthcare settings, they may also question the usefulness or have low perceived self-efficacy to perform them. To gain a better understanding of how these multiple factors interact to influence supervisors’ intention to perform direct observations, we conducted an exploratory quantitative study (Chapter 4), using the theory of planned behaviour (TPB) as our theoretical framework. In doing so, we transferred a psychological theory to medical education to obtain insight into the influence of cognitive and emotional processes on intentions to use observations in workplace-based learning and assessment. We developed a questionnaire to investigate supervisors’ intention to perform observations. The relationships between the TPB measures were explored by computing bivariate correlations using Pearson’s R tests. We performed a hierarchical regression analysis to assess the impact of the respective TPB measures as predictors of the intention to perform observations.

In this study, 82 general practice supervisors completed the questionnaire. We found that supervisors had a positive attitude towards observations. Our TPB model explained 45% of the variance in supervisors’ intentions to perform them. We found that supervisors’ normative beliefs, that is, the perceived expectations by other stakeholders in WBA to engage in performance observations, significantly predicted their intention. Consequently, supervisors’ intention to perform observations might rely heavily on the expectations articulated by other stakeholders (e.g., trainees, colleague supervisors and the training institute). Furthermore, supervisors’ past behaviour was a significant determinant of their intention to conduct performance observations, indicating that positive past experiences with observations encouraged supervisors to perform them, while negative experiences discouraged such engagement. These findings therefore suggest that supervisors use their past experiences to form intentions to perform observations in a careful, thoughtful manner; in doing so, they also take the preferences of the trainee and other stakeholders potentially engaged in observations into consideration. By discussing and clarifying underlying preferences, goals and beliefs, supervisor and trainee may be able to overcome potential barriers to the use of observations and develop a shared understanding of the role of observations in learning and assessment. In addition, we recommended that training institutes articulate their expectations regarding supervisors’ engagement in observations more clearly, explicitly and consistently. By translating their normative expectations into ongoing training and coaching of supervisors and trainees, training institutes may effectively contribute to supervisor intentions to actually enact performance observations.

Supervisors and trainees may have similar, but also diverging goals and preferential strategies for achieving these goals when engaging in observations. If their conceptions of how to use observations are misaligned, suboptimal use of such observations may result, thereby inadvertently impacting on training outcomes. The aim of our second study on the use of performance observations (Chapter 5) was therefore to better understand if and how supervisor-trainee dyads discuss and align goals and approaches to observation. We used a qualitative, social constructivist approach to explore how supervisory dyads work towards alignment of goals and approaches to observation of clinical performance. We performed a template analysis of our data, for which we iteratively created a series of templates consisting of hierarchically structured schemas of coded themes. We collected our data from a series of semi-structured interviews in eight general practice settings. In each practice, we interviewed the trainee, the supervisor and the trainee-supervisor dyad, respectively. In the two individual interviews, we explored supervisors and trainees’ preferences and the way they perceived individual goals and approaches regarding the use of observations. In the final interview, we asked the dyad to summarise their individual and perceived common goals and approaches to the use of observations, before inviting them to reflect on their viewpoints and how they worked towards alignment.

It seemed that dyads mainly engaged in such observations to guard patient safety and to prepare for institutional assessment rather than to provide the trainee with developmental feedback. It was uncommon for them to communicate explicitly about the goals and approaches to these observations, except at the start of training and unless they were triggered by internal or external factors that necessitated this discussion. Nevertheless, supervisory dyads did manage to build a working repertoire regarding the use of observations during the training year. Hence, the findings from this study seem to suggest that there might be a mismatch between the way supervisory dyads use observations and the purposes espoused and communicated by educators and/or national training institutes. Supervisor-trainee dyads must be aware of the ways they communicate about the use of performance observations in residency training, and their potential impact on the creation of a shared working repertoire. Embedding facilitated discussion and alignment of the various goals and approaches to observations in learning conversations may be a promising avenue to improve their effectiveness in workplace learning. Our findings also clearly demonstrate that the assessment programme in which observations are embedded may influence their actual use. Therefore, to foster learning from workplace observations, all stakeholders involved must be aware of the impact of institutional assessment requirements and clearly and continuously articulate the goals of assessment.

In our general discussion (Chapter 6) we synthesise the research from the previous chapters and relate our findings to current medical education literature. We describe how the conceptions of workplace-based stakeholders (i.e., supervisors and trainees) influence their approach to two important WBA strategies, namely the entrustment of clinical tasks at the very start of medical residency training and observations of trainee performance throughout training. Based on the research presented in this dissertation, we argue that stakeholder positions on the continua described in Chapter 2 may shift under the influence of various contextual factors and, as such, are dynamic rather than fixed. Our findings suggest that key elements in WBA (i.e., user conceptions, assessment tool, clinical task, purpose of assessment, educational relationship, learning environment) affect the interpretation of performance (in terms of competence and entrustability) and, consequently, WBA outcomes (i.e., providing feedback, making judgements on competence development and entrustment practices).

Our studies clearly demonstrate that, according to the stakeholders involved, trainees can be allowed to self-regulate their learning on condition that safe patient care is guaranteed. Our findings add to this that supervisors’ decision to entrust clinical tasks (Chapter 3) and/or their initiative to perform observations (Chapters 4 and 5) are likely influenced by their idiosyncratic and contextualised appraisal of trainees’ willingness and capability to ask for feedback or help in time. Hence, contextual factors may allow, and sometimes even require, supervisors to adopt a holistic, less standardised approach when making early entrustment decisions. We found these contextual factors to be equally important in mediating supervisor-trainee dyad preferences regarding performance observations. The standardised institutional consultation test, for example, although primarily aimed at promoting trainees’ development by providing relevant feedback, drove dyads’ preoccupation with ticking off items that were pertinent to the test when performing observations. After passing the test, however, the dyads gradually started to look for learning opportunities for the trainee as the focus of their observations. With this, our research illustrates that contextual factors (e.g., the perceived purpose of the assessment) may require stakeholders to (temporarily) adapt their position on the standardisation continuum. This adaptation may, however, have a negative impact on stakeholders’ perceived utility of assessment instruments and procedures and the effectiveness of WBA.

Across our studies, we identified several recurring, constituent elements of stakeholder conceptions. First, we found their conceptions of normative expectations about WBA practices to influence their WBA strategies. Within the clinical workplace, supervisors and trainees considered both mutual expectations and the perceived expectations of their patients. Moreover, the perceived expectations of the training institute, for example, affected their conceptions of WBA in the clinical setting and their associated strategies. Second, supervisors’ propensity to trust and their previous supervisory experience was another element of stakeholders’ conceptions found to recur across our studies. Our findings seem to suggest that supervisors’ starting position on the ‘desired locus of regulation’ continuum is influenced by their propensity to trust as well as their past experiences with WBA. They indicate that supervisors in the complex context of general practice training use their past experiences to decide on WBA strategies, for instance on whether and how to entrust clinical tasks or perform observations.

After providing reflections on the methodology of the research presented in this dissertation, we discussed the theoretical and practical implications of our findings. Our research findings clearly confirm that performance assessment is always interpreted and constructed according to the conceptions of its individual stakeholders who are situated in and affected by a unique social context. An important theoretical implication of our research is therefore that any approach to assessment, assessment research, and assessment design should be adapted to embrace their context. To arrive at a more comprehensive and coherent picture of a trainee’s competence development, we should acknowledge variation in conceptions of assessment between stakeholders and allow for various, contextualised assessment approaches so as to foster meaningful variance in performance interpretation. To improve the effectiveness of assessment activities and to stimulate the development of a shared understanding of WBA, ongoing awareness and articulation of one’s own conceptions of WBA, including alignment with those of other stakeholders are essential WBA strategies.

In conclusion, this dissertation offers important insights that can improve our understanding of stakeholder conceptions of WBA and their impact on the assessment strategies used during postgraduate medical training. Ongoing interaction and explicit communication between all stakeholders are essential to monitor, adapt and improve assessment practices.