A recent psychological study (Hogan et al. 2023) found that intensive Sistema programs in the US brought no additional gains in executive functioning, self-perception, or attitudes toward school in young children, when compared to general music education (or classroom music). This finding is not a major surprise, given that it echoes an earlier Sistema study from the US (Habibi et al. 2018) as well as the Inter-American Development Bank’s 2017 mega-evaluation of El Sistema in Venezuela (Alemán et al. 2017). Nevertheless, it is significant, in that Sistema programs tend to be very expensive in comparison to general music education, and so they would need to demonstrate considerable additional benefits if they were to be justifiable on extrinsic grounds. Also, Sistema’s initial expansion in the US (and elsewhere) in the late 2000s and early 2010s rested on claims of superiority to classroom music.

However, this study has passed almost entirely under the radar, as far as I can tell. I only discovered it thanks to an automatic alert from ResearchGate, which appeared because the article cited one of my own. None of the other Sistema-watchers that I know had seen it. I cannot find any evidence of others having written about it or cited it. This reflects a point that I made in my 2021 book Rethinking Social Action Through Music (244-45), about the bias towards positive findings and inflation of effects in the production of knowledge about music education in general (and specifically about Sistema):

“research studies themselves do not generally articulate grandiose statements about miracles and social transformation; rather, some point to small cognitive or psychological differences and benefits, while others do not. The two largest randomized controlled trials (RCTs) in this field found no effect of music training on cognitive or academic skills (Haywood et al. 2015; Alemán et al. 2017). But in the translation into the public realm, many caveats and limitations are ironed out, and null or negative findings are generally overlooked, since there are no organizations in whose interest it is to promote them. As Sala and Gobet (2020) note, the two major RCTs above have been paid little attention by the media or even by other researchers, even though RCT is the gold-standard methodology. More positive studies are more likely to be picked up by advocacy organizations and to lead to a report in the media, in which small-scale and specific findings often become an expansive and generalized story about the power of music (Mehr 2015; Odendaal et al. 2019). Many musicians encounter headlines, summaries, and animations of such stories on social media. As a result, there is a significant gap between the mixed findings and caution of some researchers concerning the transfer effects of music education, and the more uniformly optimistic opinion that prevails among musicians and the general public (Mehr 2014; D’Souza and Wiseheart 2018).

There is virtually no appetite for questioning the dominant narrative in the public sphere, meaning that counter-arguments are rarely heard. The classical music industry and profession have lined up behind a story that benefits and flatters them, and many classical music journalists have followed suit. Few are willing to risk arousing the wrath of music lovers by presenting less positive research conclusions to a wider public [for example, by writing an article for The Conversation]. The ambiguous picture presented in these pages will not be a surprise to many researchers in fields such as the sociology of music education or development studies, who are used to confronting counter-effects, unintended consequences, and gaps between aims and outcomes. In the public sphere, though, ambivalence about the power of music education is a rare bird.”

(I also address the issue of bias in this more recent blog post on the book The Power of Music.)

Had the recent study produced positive results, an international network of Sistema programs, advocates, and supporters would have amplified the story and incorporated it into their publicity. But who is going to spread the word about a null finding? Only positive outcomes impinge on the public consciousness and become part of the dominant narrative. As a result, public understanding is skewed.

As I argued in a recent article, Sistema is propelled by ideology, not science. The Venezuelan original took the form of classical youth orchestras because it was created by an orchestra conductor who loved classical music; all the claims about “social action through music” arose much later, without evidential grounding, as justification for devoting ever-increasing sums of public money to funding its founder’s private passion. El Sistema’s official mission and vision revolve around moral and spiritual claims, not scientific ones: “the ethical salvation of children and young people,” “the comprehensive development of human beings,” “transcendental values,” “rescuing children and young people from an empty, disorientated, and deviant youth,” and so on. The Sistema boom in the global North from 2007, spearheaded by the likes of Simon Rattle, was driven by perceptions or hopes that it represented a resurrection of classical music, and a variety of backers jumped on board in pursuit of a range of  cultural, political, and economic objectives.

Part of its adaptation to its new contexts, though, was a requirement to play the game of evidence-based decision making. So the ideological motivations were soon covered by evaluations making all sorts of scientific(-sounding) arguments about the distinctive psychological and cognitive benefits conferred by Sistema, and skirting around the bigger questions that it raised. For a while, what Owen Logan called “Sistema-friendly research” provided a fig-leaf for the sector’s ideological drive. But the IDB’s 2017 evaluation of El Sistema was a big nail in the coffin for such justifications: after all, if the program’s own funder (in the biggest evaluation yet carried out, focused on the original version) could not find convincing evidence of cognitive or prosocial effects, then all such claims looked distinctly shaky. It was soon backed up by Habibi et al.’s study of a Sistema program in the US. Now, a few years later, comes this new research by Hogan et al.

Yet such studies have been largely ignored by the sector and its funders, revealing the performative nature of the whole evaluation enterprise. The game is indeed a game. One might be excused for thinking that evaluation provides the evidence for evidence-based decision making, but in reality the key decisions have already been taken (on ideological grounds) and evaluation is supposed to provide post hoc justification for them, hence anything that questions the status quo is ignored or spun. The IDB’s discovery that El Sistema didn’t work did not lead to defunding or radical changes, but rather to efforts to present the troublesome findings in a rosier light and to carrying on as before (see Baker, Bull, and Taylor 2018).

This is hardly unique to Sistema – as Ben Ramalingam (2013, 10) notes in his study of development:

“When the political context is not right, research is bypassed, evaluations are forgotten, [and] studies are ignored. […] Numerous policy evaluations and research studies […] end up wrecked, abandoned, or disappearing altogether.”

He quotes an anonymous donor: “We say we want evaluation, but we don’t, we want results, results we can put in our  glossy reports, we can put on our websites that we can give to ministers to present in Parliament” (120).

(El) Sistema provides a good illustration of Ramalingam’s point. So where does that leave all this evaluation of Sistema programs?

On the one hand, I’m glad that the present study was done – it is interesting to see further evidence that the claims made by and for Sistema are exaggerated. Such evidence is potentially important at the level of program design and educational policy. On the other hand, it’s a bit of a waste of time if only a handful of researchers like me are going to pay any attention to it.

Just imagine if all the money and effort that has gone into 15 years of quantitative studies on (El) Sistema, into building a narrative on positive findings and excusing or ignoring null or negative ones, had instead been focused on exploring the latest qualitative research on pedagogy, curriculum, culture and development, etc. and on grappling with the question posed by Gert Biesta: what constitutes good education, and not just effective education?

Effects can potentially be produced by ethically and culturally questionable means. The Third Wave, an experiment carried out at a California high school in 1967 by a history teacher named Ron Jones, produced immediate improvements in discipline, motivation, and academic achievement – via an experiment in fascist-style education. José Antonio Abreu believed in the pursuit of musical excellence by any means necessary, and research going back more than 25 years has demonstrated that those means were often not pretty. In some hands, El Sistema was the very antithesis of a good education.

So much focus on effects can marginalize necessary debate about the crucial how of education. It is at the level of process that music educators, researchers, and funders should be focusing their efforts. If the process is right, if the education is good, the results will take care of themselves.

References

Alemán, X., Duryea, S., Guerra, N. G., McEwan, P. J., Muñoz, R., Stampini, M., & Williamson, A. A. (2017). “The effects of musical training on child development: A randomized trial of El Sistema in Venezuela.” Prevention Science, 18(7), 865–878. https://doi.org/10.1007/s11121-016-0727-3

Baker, Geoffrey, Anna Bull, and Mark Taylor. (2018). “Who Watches the Watchmen? Evaluating Evaluations of El Sistema.” British Journal of Music Education 35 (3): 255–69. https://doi.org/10.1017/S0265051718000086

D’Souza, Annalise A., and Melody Wiseheart. (2018). “Cognitive Effects of Music and Dance Training in Children.” Archives of Scientific Psychology 6 (1): 178– 92. http://dx.doi.org/10.1037/arc0000048

Habibi, A., Damasio, A., Ilari, B., Elliott Sachs, M., & Damasio, H. (2018). “Music training and child development: A review of recent findings from a longitudinal study.” Annals of the New York Academy of Sciences, 1423(1), 73–81. https://doi.org/10.1111/nyas.13606

Haywood, Sarah, Julia Griggs, Cheryl Lloyd, Stephen Morris, Zsolt Kiss, and Amy Skipp. (2015). “Creative Futures: Act, Sing, Play. Evaluation Report and Executive Summary.” NatCen Social Research. https://e-space.mmu. ac.uk/618917/1/Act__Sing__Play.pdf

Hogan, J., Cordes, S., Holochwost, S., Ryu, E., & Winner, E. (2023). “Participation in Intensive Orchestral Music Training Does Not Cause Gains in Executive Functioning, Self-Perception, or Attitudes Toward School in Young Children.” Psychology of Aesthetics, Creativity, and the Arts. Advance online publication. https://dx.doi.org/10.1037/aca0000593

Mehr, Samuel A. (2015). “Miscommunication of Science: Music Cognition Research in the Popular Press.” Frontiers in Psychology 6: 988. https://doi.org/10.3389/ fpsyg.2015.00988

Odendaal, Albi, Sari Levänen, and Heidi Westerlund. (2019). “Lost in Translation? Neuroscientific Research, Advocacy, and the Claimed Transfer Benefits of Musical Practice.” Music Education Research 21 (1): 4–19. https://doi.org/10. 1080/14613808.2018.1484438

Ramalingam, Ben. (2013). Aid on the Edge of Chaos. Oxford: Oxford University Press.

Sala, Giovanni, and Fernand Gobet. (2020). “Cognitive and Academic Benefits of Music Training with Children: A Multilevel Meta-Analysis.” Memory & Cognition 48 (8): 1429–41. https:// doi.org/10.3758/s13421-020-01060-2