Measuring populism: content analysis

Several scholars define some parties as being populist and other parties as not being populist. They make use of a minimal definition (Sartori, 1970) in which they set out a number of minimal criteria which a party must meet in order to be labeled populist. Mudde (2007), for instance, classified an impressive number of populist parties in both Western and Eastern Europe by means of this approach (see also Vossen 2009 for a similar approach of minimal criteria). Although this method constitutes a good starting point, it is difficult to apply systematically in practice as the method is rather subjective; often it is not entirely clear why some parties are populist and others not. Moreover, some parties - such as the LPF - do not entirely fit the definition of Mudde (Lucardie 2007), even though they are labeled populist. Parties such as the LPF might be mildly populist, a nuance the minimal definition does not allow for. This is why we opt for measuring populism by means of a scale. The “black-and- white approach” in which parties are defined as either populist or not is explicitly abandoned because it misses various shades of grey. By adopting a “matter of degree” approach we hope to overcome the over-simplified dichotomous “either-or” thinking that has plagued research on populism.

In this paper we measure populism by means of the method of content analysis. The reason is that content analysis is an appropriate technique for measuring the ideas of political parties in different countries and different time periods.^[68] In this section we discuss two recent papers that have measured populism by means of a content analysis. After an overview of these studies, we present the most important differences with our own approach and illustrate the two specific content analysis methods with which we have measured populism in this paper: the classical approach and the computer-based procedure.

Jagers and Walgrave (2007) were among the first scholars to produce a systematic quantitative empirical analysis of populism. They define populism as a political communication style and measure it by means of an analysis of political party broadcasts of Belgian (Flemish) political parties. Although Jagers and Walgrave’s analysis formed a breakthrough in measuring populism, it is limited to one country only (actually only Flanders), and it is not entirely clear whether it would work as well for other data than the rather scarce political party broadcasts. Moreover, issues of reliability and validity are not dealt with. Another scholar who conducted a content analysis to measure populism is Hawkins (2009). He conceptualizes populism as a political discourse and measures it by means of “holistic grading” of speeches by chief executives in an impressive number of mainly Latin-American countries. Holistic grading means that the unit of analysis is the entire text. The author trained native speakers to analyze speeches according to a rubric that, in his opinion, captures the core elements of the populist discourse. Speeches were ranked on a 3- point scale (nonpopulist; mixed; populist). The main problems with Hawkins’ study are the rough - and therefore possibly invalid - measurement due to the holistic grading method, and at times low reliability (Kappa = 0.44).

Our approach differs in at least two important respects from the studies of Jagers and Walgrave and Hawkins. First, we have defined populism neither as a style nor as a mere discourse. As argued before, we perceive populism as a thin ideology. This focus on populism as a mainly substantive and ideological phenomenon, brought us to decide to analyze election manifestos instead of political party broadcasts or speeches. We have chosen to focus on manifestos for two reasons, one substantive and one more practical. The substantive reason is that an election manifesto can be seen as the document that gives the clearest overview of where a party stands for at a certain point in time. The more practical reason is that election manifestos are appropriate documents for a cross-national content analysis, because they are reasonably comparable across countries and over time. A disadvantage of election manifestos is that most voters never read these documents. Other party sources, such as for instance newsletters, are much more widely read. Contrary to election manifestos (which are often unexciting enumerations of ideas and plans), such sources are usually more vividly written texts. It might therefore well be argued that in such sources the dimensions of populism are more accentuated than in the “dull” election manifestos. However, because we have defined populism as an ideology, and not as a style or a discourse, we believe that we should focus on the documents that most clearly present the political ideas of political parties: the election manifestos.

Second, we pay extensive attention to issues concerning validity and reliability. Jagers and Walgrave neglect both, while Hawkins pays attention only to reliability. Validity concerns the question whether we measure what we think we are measuring. This is crucial for a slippery concept such as populism. Reliability means that the analysis must be consistent. This implies that repeated measurements should lead to the same results (King, Keohane and Verba 1994: 25). Although validity and reliability are of fundamental importance for every measurement in social science, it plays a maybe even more pivotal role in content analysis. “To stand on indisputable ground, content analysts must be confident that their data (a) have been generated with all conceivable precautions in place against known pollutants, distortions, and biases, intentional or accidental, and (b) mean the same thing for everyone who uses them” (Krippendorff 2004).

We have measured the thin ideology of populism with two different content analysis methods: a classical content analysis and a computer-based content analysis.

Дата добавления: 2015-07-10