Discussion: the trade-off - suggestions for future research

In many cases there exists a trade-off between validity and reliability (Goldenberg 1992). To what extent the one should prevail over the other depends on the research design. We believe that in large-n comparative studies of populism it might be a good idea to focus on reliability and use a computer-based analysis. We have seen in this paper that a computer-based content analysis of populism can yield valid results. We have to be very careful, however, with the interpretation of the outcomes. Populism is highly sensitive to specific contexts and therefore difficult to “grasp” and measure automatically. This would be much easier in a classical content analysis. Yet a classical content analysis in a large-n study would be very time­consuming and expensive. More importantly, it will be very difficult to get the analysis reliable, because of the strong contextual differences between countries. Though a computer- based content analysis may come at the expense of validity, one is at least certain of sufficient (read perfect) reliability scores.

In a more qualitatively oriented case study it might be a better idea to measure populism by means of a classical content analysis. Especially if the goal is to gain deep understanding of specific parties or politicians, it is more appropriate to focus in detail on the dimensions of people-centrism and anti-elitism. How exactly does a political actor perceive “the people”, and how does it express its negativity towards elites? These important questions are nearly impossible to answer with a computer-based analysis. Human coders are needed to interpret contextual differences.

For medium-n comparative studies on populism (n = 3-7), we propose a combination of the classical approach and the computer-based method: an integrated man-machine approach that filters out the disadvantages and emphasizes the strong points of both methods. This method can be applied to every kind of text (spoken or written): from speeches to election programs and from newspapers to parliamentary debates. The following procedure would yield both valid and reliable results. First, every word that could refer to the people or to negativity towards elites is included in a dictionary. Second, a computer scans the relevant texts for all the selected words. Third, human coders interpret the results of this computer analysis by means of a codebook. Focusing on the selected words they decide for every paragraph whether there is a reference to the people and whether the author of the text is negative towards elites. This approach will generate more reliable results than a classical content analysis, because the first selection is made by a 100 per cent reliable computer. Moreover, it will also be much faster and therefore less expensive. The approach will also produce more valid results because the first selection of the computer will be double-checked by human coders who are able to interpret the contextual meanings of the selected words.

Populism is a complicated concept. But so are most concepts in social science. Conceptual complexity should not stop us from trying to find valid and reliable methods to measure them.

Дата добавления: 2015-07-10