When I saw that Susan Hallam and Evangelos Himonides’ monumental new book “The Power of Music: An Exploration of the Evidence” had been published, I went straight to my area of specialism – El Sistema – to see how they handled it. It turned out to be quite a prominent theme in the book; a search showed the word “Sistema” appearing 95 times. But I quickly discovered numerous problems with its treatment.
The authors seem to have gone out of their way to present a positive image of El Sistema. They omit any reference to most of the critical literature on this subject. Only a single article is presented as raising questions about El Sistema. Neither of the two academic monographs on this topic are included. Studies with mixed conclusions are cited, but only their positive findings are mentioned. For example, two of Gabriela Wald’s articles appear, but her arguments about social division and reproduction are ignored, along with her evidence that students dissented from dominant narratives of social action through music. The authors give no inkling that Wald is a critic of Sistema-style programs in Argentina. Provenzano et al. (2020) is present, but only its positive results are cited; there is no mention of the fact that “program participants reported experiencing a decrease in satisfaction with life from preintervention to postintervention” or that the researchers identified this finding as “a major limitation of this study” (three other significant limitations are also listed). Furthermore, the same authors’ earlier article (Hopkins, Provenzano, and Spencer 2017), which is considerably more critical of El Sistema, is not included.
The most substantial quantitative study (Alemán et al. 2017) is repeatedly cited without any reference to the obvious question marks over its methods, to published, critical discussions of its findings (Baker, Bull, and Taylor 2018; Clift, Phillips, and Pritchard 2021), or to its damning conclusion that El Sistema did not reach more than a small number of poor children. The study’s authors acknowledged that El Sistema illustrated “the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program.” As Stephen Clift (2019) has noted in reference to this study: “As poorer children were under-represented, far from addressing social inequalities, the work of the [El Sistema] centres served to reinforce them – entirely contrary to the idea of an intervention designed to reduce social and health inequalities.” But Hallam and Himonides are silent on this major finding that undermines all the other claims about the Venezuelan program. Worse still, they assert that El Sistema fosters prosocial behaviour when Alemán et al. stated clearly that they “did not find any full-sample effects on cognitive skills… or on prosocial skills and connections.”
And the list goes on. The authors use older literature reviews by Creech et al. (2013; 2016) rather than more recent ones by Puromies & Jovonen (2020) and Bolden (2021) in which more critical literature is surveyed or more ambivalent conclusions are drawn. They include an article built on media reports, documentaries, advocacy literature, and institutional publicity materials – in other words, one that has no place in this kind of book – and another based on fieldwork by a student with no research training. The book’s position on the Sistema-inspired field rests heavily on program evaluations, without any discussion of their limitations as evidence or their potential skewing towards positive conclusions. This is a topic about which there has been plenty of academic debate: Jancovich and Stevenson (2021), for example, raise questions about the credibility of evaluations, arguing that many “reinforce overstated celebratory narratives,” and Baker, Bull, and Taylor (2018) illustrate this point specifically in relation to evaluations of El Sistema. The first factual sentence on El Sistema (p.326) contains two errors: the name of its founder is wrong, and the authors incorrectly claim that El Sistema was created as social action through music (a term that was actually adopted decades later, as I have noted in several academic publications).
In short, the book shows not just errors but also an overwhelming bias in its treatment of El Sistema. Bias is not a problem per se – some would argue that there is no such thing as unbiased research – but it is a problem when it is not acknowledged. It is a real problem in a book that presents itself as an objective review of the evidence, rather than a work with an explicit stance.
Positionality statements, acknowledging and explaining potential biases, have become commonplace in the social sciences, and are little short of a requirement in PhD theses these days, yet there is no sign of one here. On the contrary, the book makes implicit claims to neutrality. It states: “An inclusive research strategy was adopted in accessing the literature to be included in this book,” and its subtitle is “an exploration of the evidence” – yet some of the most substantial evidence has been excluded. Its preface ends: “It is hoped that […] the reader will be enabled to make an informed decision about the power of music in a range of areas across the lifespan.” How can the reader make an informed decision when the authors withhold some key sources of information and misrepresent others? The authors have a partial view, and they give the unsuspecting reader no choice but to adopt it. This sleight of hand – advocacy disguised as objectivity – is distinctly troubling in what will undoubtedly become a cornerstone text.
To be clear, we are not just talking about a little bit of gentle favouritism here. Ten years of heated debate in books, articles, special issues, book chapters, conferences, newspapers, magazines, radio, TV, and social media are reduced to: “Not all of the research on Sistema-inspired projects has been positive.” Bibliographic leads for any reader intrigued to know more are almost entirely lacking. The authors’ claim about prosocial skills is squarely the opposite of the conclusion of the report that they cite – the academic equivalent of claiming that black is white. All academics make mistakes, but taken in the context of the treatment of El Sistema in the book as a whole, this is something else: Hallam and Himonides’ judgment appears to have been completely clouded by their advocacy stance.
I am reminded of Clift and colleagues’ recent critiques of large-scale surveys on the role of the arts in improving health and wellbeing, for losing sight of the crucial issue of the quality of the evidence in their focus on quantity. (In addition to the articles cited above, see here.) As we see above, there are several areas in which quality can be compromised in such surveys: in the studies that are cited (they may not be robust); in the way that they are cited (lack of critical scrutiny, selective reading, misrepresentation); and in the studies that are not cited (the omission of relevant but complicating evidence).
Another area is highlighted in a further article by Clift’s team: “vested interests” of academics performing the review. They ask: “Is the starting point of a review team one of ‘dispassionate enquiry and scepticism’ or is there a pre-established conviction […]? If the former, a review team may interrogate research methods and findings closely in the interests of establishing the truth or otherwise of claims made. If the latter, a review may be undertaken with the purpose of showcasing positive evidence.” This point is highly relevant in our case. Hallam and Himonides seem determined to sing the praises of El Sistema despite the many criticisms it has faced from other researchers, its obvious disjunctures with contemporary thinking on music, development, and social justice, and the string of public scandals (including repeated allegations of widespread sexual abuse) that undermine claims that it is a positive exemplar of “the power of music.” Why do the authors adopt this perplexing stance?
Rimmer (2020) provides important clues. It turns out that Hallam has advocated for the El Sistema model since 2010 and has been connected to the Sistema-inspired field in various capacities, including as a program evaluator and research supporter. Her husband was a driving force behind Sistema England and a member of its advisory board. Rimmer concludes that Hallam formed part of an “inner circle” and a “network of sympathetic voices” that served as key “sponsors” of Sistema in England. Hardly “dispassionate enquiry and scepticism,” then. Hallam’s vested interests in Sistema provide an obvious explanation for the bias in her review, including its ample, uncritical use of program evaluations. Rather than a neutral, inclusive exploration of the topic, as is claimed, the book is a prime example of what Owen Logan (2015) calls “Sistema-friendly research”: studies that “dovetail with the ‘spin’ coming from Sistema projects internationally, to the extent that researchers would appear to have a hand-in-glove relationship with the Sistema network.”
This slice of the book is highly anachronistic: rather than reflecting the lively academic debate on El Sistema that has emerged since 2014, it echoes the monochrome, ill-informed enthusiasm of a decade ago. It is a throwback to Hallam’s advocacy for Sistema England in the early 2010s – before independent research had begun to raise awkward questions, and long before a sexual abuse scandal in Venezuela led Sistema England to distance itself from the mother program (it closed down just months later). But ignoring the last eight years of vigorous and very public debate is no way for a literature review to proceed. Researchers of music education and social change should be perturbed that such a major work advocates for a program and a model that have been much questioned in recent years, and that it does so in such a problematic way. This book is a backwards step for El Sistema research and practice, and with its endorsement of the status quo, it may constrain much-needed reform and rethinking.
The fact that this problematic presentation of El Sistema comes from one of the most lauded figures in the music education field also raises uncomfortable questions about the reliability of academic discourse on the social impact of music. To be sure, I have examined just one sliver of the book, though with 95 mentions, it’s a significant one. But a question that emerges from Clift et al.’s work in this field is that if we take one element of a wide-ranging review and find it flawed, then how can we have confidence in the review as a whole or in its higher-level conclusions? If quality is compromised at a granular level, then the whole edifice is shaky, however impressive it may seem at first glance.
How many other hidden agendas are woven into the text, how much more of the evidence consists of sub-standard studies, and what else has been partially reported or omitted for not fitting with the story that Hallam and Himonides wish to tell? Answering that question in a comprehensive manner would be an unfeasibly large task, and maybe an impossible one, since a thorough knowledge of the literature on any given topic is required in order to spot potential bias. But the section on “Music and victims of abuse” deals only with music as a form of treatment or therapy; there is no mention of the well-known problem of sexual harassment and abuse within music education institutions. El Sistema makes an appearance in this section too, but (of course) only in a positive light; nothing is said about the abundant evidence of abuse of various kinds within the Venezuelan program, which has been reported in several academic publications and numerous international media outlets. The chapter on “Psychological wellbeing” has little to say about musicians’ wellbeing and the risks and emotional harms of musicianship for some, as eloquently discussed by Musgrave (2022). The section on “Social inclusion” makes no reference to critiques of this concept and its attachment to culture and education. In other words, there are signs that El Sistema is not a unique case in this book, and that critical questions and complicating evidence concerning negative aspects of music and music education may have been downplayed or elided in a more consistent fashion. If that is the case, then readers will have to look elsewhere in order “to make an informed decision about the power of music.”
Geoff Baker 
Catching up with your recent work, Geoff. This is a damning assessment, and surely raises questions for the editorial team involved too. Have there been any attempts by Hallam to answer the points you make here? Once again, this seems to demonstrate the extraordinary power of orchestral music allied with emotive messages of social uplift to lead many well-meaning people to switch off their critical faculties. Silence on the part of El Sistema advocates, and those who share their biases, consciously or not, simply underlines the need for a robust and open debate.
LikeLike