Annemarie Neeleman

6.2 Summary

This section summarizes the research aims, methodological approaches, and main findings of each of the four studies.

6.2.1 The construction of an empirically based classification of school interventions and the application of this classification to the distribution of current Dutch secondary school interventions (Chapter 2)
This study, first, presented the construction and validation of an empirically based classification of school interventions that allows for the identification, analysis, and comparison of the actual exercise of school autonomy. The classification is organized via three main domains: education, organization, and staff. Each of these domains consists of various subdomains, with the entire framework composed of 16 such subdomains. The construction is based on the digital questionnaire responses of 196 Dutch secondary school leaders reporting a total of 735 school interventions. Due to the high level of school autonomy in the Netherlands, school leaders have decision-making authority in many areas. In a deliberate attempt to grasp the full potential range of actual school interventions, the researcher formulated the question on school interventions in an open-ended manner. For the same reason, a school intervention was broadly defined as a planned action intended to cause change in the school. To achieve practical (i.e., face and content) validation of the classification, the researcher actively involved school-level decision-makers in all stages of the study. Consequently, the classification can capture a wide range of school interventions; has enough depth and detail to allow for meaningful distinctions; and features a logic and structure to enhance wide usability at the local, national, or interventional level by practitioners, policy-makers, training institutes, and researchers alike.

Using the developed classification, this study secondly presented two distributions: one involving the 595 school interventions that the school leaders had introduced or were about to introduce, and one comprising the 140 interventions that they had seriously considered but not introduced. Both distributions highlight the dominance of educational interventions: close to half of all introduced and considered interventions were educational in nature. Organizational interventions were, in turn, more frequent than staff interventions. There was a clear dominance of 4 of the 16 subdomains: educational programs, learning environments, pedagogical approaches, and professional culture. These four subdomains—three of which are in the educational domain—comprised nearly 60% of all school interventions that the Dutch secondary school leaders either pursued or considered pursuing. The three most frequent interventions were digital tools and methods for teaching and learning, peer professionalization, and differentiation interventions.

6.2.2 The relationship between actual school interventions and factors found in educational effectiveness syntheses (Chapter 3)
This study analyzed how current Dutch secondary school interventions relate to the effectiveness factors presented in three internationally authoritative educational effectiveness syntheses: Robinson et al. (2009), Scheerens (2016), and Hattie (2009). Meta-analyses are comprised of multiple individual rigorous studies, and as such, they present robust results regarding the effectiveness of the items in question. Since no syntheses have been exclusively based on Dutch secondary education research, international syntheses were employed. To analyze the interventions from both a school perspective and a school leadership perspective, the researcher used syntheses from both effectiveness traditions. The different aggregation levels of the effectiveness factors were accounted for, and effect sizes and ranks were included when available.

The comparative analysis resulted in five main findings. First, the vast majority of analogies between school interventions and effectiveness factors were general in nature. This means that the analogies were so abstract that they were not highly informative. Second, a few school interventions paralleled relatively specific factors from Scheerens (2016) and, particularly, Hattie (2009). The factors presented in Robinson et al. (2009) all had a general character. Third, a diverse range of school interventions, and especially those in the organizational and staff domains, lacked an analogous effectiveness factor altogether. Fourth, the mean effect sizes calculated for those interventions analogous to effectiveness factors indicated that across all syntheses, current Dutch school interventions tended to be similar to factors with relatively low—or even negative—effect sizes. Fifth, the more detailed analysis of the three most frequent interventions suggested that the three syntheses used for this comparative analysis provide little evidence of these particular interventions notably improving student achievement. All in all, the comparative analysis demonstrated that a wide range of current Dutch secondary school interventions lack an analogous factor in one or more of the examined syntheses, despite the relatively inclusive stance adopted in identifying those analogies. Additionally, the mean effect sizes, along with the findings regarding the three most frequently implemented interventions, demonstrated that the three syntheses offer little evidence that the vast majority of interventions substantially improve cognitive student achievement.

6.2.3 School leaders’ personal beliefs and the motives behind their school intervention decisions (Chapter 4)
This study aimed to obtain a better understanding of the motives behind Dutch secondary school leaders’ school intervention decisions. To this end, 10 individual semi-structured interviews were conducted with school leaders who were purposefully sampled based on the strategy of maximum variation sampling. A literature review showed that school leader behaviors and actions tend to be influenced by a great many, often interlinked, factors at the personal, organizational, and societal levels. Because of the observed dominance of personal beliefs in this interplay in some studies, this concept was included in the research questions that guided this study. In an attempt to make the topic of school intervention decision-making concrete enough for school leaders to share their practices, the researcher applied the criterion of specificity. To both connect with school leaders’ current intervention preferences and enable a potentially large sample, the researcher selected for this purpose a specific intervention that appeared frequently in the questionnaire: differentiation. At the end of the interview, school leaders were asked to score a list containing factors at the personal, organizational, and societal levels that potentially affect school intervention decisions. The factor list was included as a form of data triangulation to increase the validity of the interview findings.

Discussed using a remarkably similar vocabulary, Dutch school leaders’ personal beliefs refer to connecting and collaborating with others, a search for moral purpose and significance, and the need to facilitate talent development as well as well-being and a safe learning environment. These shared core beliefs convey a strong, value-driven, holistic, people-centered orientation with an emphasis on relationships with students and colleagues and their development and well-being. The reasons the school leaders chose to initiate a differentiation intervention proved to be closely related to their personal beliefs. Rather than being motivated by the (explicit) ambition to improve cognitive student achievement or to follow research evidence, differentiation was predominantly motivated by the school leaders’ beliefs about the pedagogical task of education. Three of the four prevailing motives were related to providing education that is tailor made and directed at talent development, that motivates and activates students, and that prepares them for their future roles in a changing society. The fourth dominant cluster of motives arose from the need to survive as a school. The school leaders attached great importance to the school’s distinctive profile and image among neighboring schools so as to ensure student enrolment in an education system with school choice and, in many areas, declining student numbers. Individual organizational and societal factors played a smaller role in the school leaders’ decision-making concerning the differentiation intervention than most personal factors. The main exception was the school mission statement, which tended to unite factors at all three levels and, as such, provided the school leaders with the organizational (school policy) foundations to pursue the particular intervention. These mission statements, in turn, closely matched the school leaders’ personal beliefs. Finally, all school leaders indicated that their personal beliefs not only influenced their decision regarding the specific differentiation intervention but guided their intervention decisions in general.

6.2.4 Evidence use by school leaders in school intervention decision-making (Chapter 5)
This study aimed to provide insight into school leaders’ actual use of evidence in their school intervention decision-making. School leaders’ evidence use was explored by means of a mixed-methods approach that combined observations from a large number and wide variety of school leaders with illustrations of school leaders’ actual use and interpretation of evidence in their decision-making practice. Evidence was defined broadly as including school data, school action research, and research evidence (not exclusively academic evidence). In a digital questionnaire, school leaders were asked to indicate per entered school intervention if they had used evidence in their considerations regarding whether to pursue the intervention. School leaders were subsequently asked to list the consulted evidence source(s) concerning one of their—randomly selected—school interventions.

In a series of semi-structured interviews, no direct questions were asked about the role of evidence in the decision-making process to avoid a potential social desirability bias. It was believed that if evidence had indeed influenced the school leaders’ intervention decisions, it would manifest itself in their accounts of their considerations.

In the questionnaire responses, 196 school leaders, first, demonstrated a very high self-reported use of evidence in their school intervention decision-making. The questionnaire, second, pointed to large differences in evidence use across the 16 school intervention subdomains. Third, although school data and action research proved to be school leaders’ preferred evidence sources, other evidence sources—such as evidence from academia, knowledge brokers, or other schools—accounted for a considerable share of total evidence use as well. Fourth, school leaders frequently used two or more evidence sources in the decision-making process concerning one intervention. Fifth, school leaders’ open-ended answers about evidence sources indicate a very liberal use and, hence, interpretation of evidence, even compared to the expansive definition that was applied.

Compared to the questionnaire findings, the interviews, first, attenuated rather than supported the high levels of self-reported evidence use. Second, the interviews underlined the liberal interpretation and use of evidence. The interviews additionally indicated that evidence use is highly dependent on personal relationships. Rather than referring to formalized evidence sources, the school leaders demonstrated evidence use that originated from personalized sources, such as professional learning networks, teachers following master’s programs, knowledge brokers, and good practices at other schools. Most school leaders used evidence conceptually at most, even with the broad definition that the school leaders themselves stretched even further. Only 1 of the 10 interviewed school leaders used evidence instrumentally. At the time of the interviews, two school leaders were searching for academic evidence to support an intervention they had already introduced. If they indeed managed to find evidence, these instances would represent cases of symbolic/political evidence use.