Evaluating behaviour change interventions delivered through mass media

Abstract

Conservationists are increasingly aware of the importance of behavior change interventions to tackle threats to biodiversity. One of the most common strategies for the dissemination of demand reduction messages is the use of mass media (e.g., radio or television). The mass media are highly appealing as they can reach thousands or even millions of people simultaneously. However, there are important barriers to the effective evaluation of these behavior change interventions. One of them is self-selection bias, which stems from the fact that people actively choose to be either part of the control or treatment groups. This bias means that comparing listeners to non-listeners, as is commonly done, is likely to yield biased estimates of impact, as those that actively choose to listen to programs containing wildlife related messages are more likely to have an interest in wildlife and its conservation. We investigate this issue through the case study of an intervention to reduce demand for bushmeat in five communities, in northern Tanzania. The intervention was centered around 25 episodes of a radio drama, part of a one-hour radio show. Each episode of the serial radio drama was accompanied by a 45-minute interactive call-in show, which featured interviews with experts, local information about community resources, and provided the audience with a platform to reflect on the drama, share opinions and ask questions. We evaluated this intervention using a Before-After-Control-Impact framework based on longitudinal data from 168 respondents. To account for potential selection bias we used a matching algorithm together with ordinal regression to build a credible control for our listener group. This was done by matching respondents on their knowledge and attitudes towards bushmeat, their community of origin and baselines values for all outcomes of interest. Our analysis did not uncover any statistically significant differences between the treatment and control groups, suggesting that the intervention was not effective. Yet, given the proximity of some of the outcomes variables to statistically significant effect sizes and the lack of statistical power arising from a small sample size, we also undertook falsification tests, which test for changes in variables that would not be expected to change as a result of the intervention under consideration (e.g., variables related to ecotourism). We found that there were indeed several statistically significant changes. This indicated that other factors may have driven behavioral changes in the target audiences and thus that it was likely that those outcomes variables approaching statistically significant values were spurious results and not the result of lack of statically power. Only through more robust evaluation of behavior change interventions and the sharing of lessons learned can conservationists successfully tackle complex issues such as the demand for bushmeat.