Understanding musical expressiveness using interactive multimedia platforms

Abstract

Background in musicology. Understanding the gesture-based foundations of musical involvement opens a number of new perspectives for musicology, with the likely effect of a change in approach. Giving justice to the role of gesture in music calls for an integrated view of perception and action, which implies a shift from auditory-based cognition towards embodied cognition, an approach that includes the whole human body as mediator between mental processes and physical energy. Background in computing. With the help of new measurement tools, new computational models of multi-modal interaction, and new interactive multimedia platforms, it becomes easier to study music from a multi-modal point of view, both in terms of scientific and artistic perspectives. Music also offers an appropriate domain for the testing and developing of interactive systems and computational models for handling non-verbal multi-modal communication. Aims. In studying musical gesture, both musicology and engineering can profit from a joint collaboration. While technology provides new tools for the measurement and modelling of musical involvement, knowledge of its corporeal foundation may largely contribute to the development of technology that in turn contributes to measurement and modelling. Furthermore, this knowledge may be exploited in artistic applications. As such, the interaction between musicology and music engineering may lead to interesting developments in the domain of embodied cognition and interactive multimedia platforms. Main contribution. An overview of recent work on the development of an interactive multimedia platform for the measurement and modelling of musical expressiveness is presented. The paper introduces the notion of corporeal imitation and mirroring behavior as a core concept fur understanding musical expressiveness. Implications. The approach advocated in this paper implies a shift of paradigm in systematic musicology, away from an all too narrow focus on auditory perception and sound structure, towards a broader view that encompasses multi-modal musical involvement. This approach is grounded in multi-sensory integration, the coupling of perception and action, the study of motor imitation, and issues that relate to affect, emotions and subjectivity. The advantage of using a interactive multimedia platform for music research, rather than ad hoc solutions, pertains to the use of modular functionalities, which allow rapid prototyping of experimental setup, flexible modelling, and the subsequent use of results in artistic applications.