Interaction of sight and sound in the perception and experience of musical performance

Abstract

Recently, Vuoskoski, Thompson, Clarke, and Spence (2014) demonstrated that visual kinematic performance cues may be more important than auditory performance cues in terms of observers’ ratings of expressivity perceived in audiovisual excerpts of piano playing, and that visual kinematic performance cues had crossmodal effects on the perception of auditory expressivity. The present study was designed to extend these findings, and to provide additional information about the roles of sight and sound in the perception and experience of musical performance. Experiment 1 investigated the relative contributions of auditory and visual kinematic performance features to participants’ subjective emotional reactions evoked by piano performances, while Experiment 2 was designed to explore the effect of visual kinematic cues on the perception of loudness and tempo variability. Experiment 1 revealed that visual performance cues seem to be just as important as auditory performance cues in terms of the subjective emotional reaction of the observer, thus highlighting the importance of non-auditory cues for music-induced emotions. The results of Experiment 2 revealed that visual kinematic cues only affected ratings of loudness variability, but not ratings of tempo variability.