Visualizing Telematic Music Performance

Telematic music encompasses a vibrant field of research and practice exploring the aesthetic, technical, and cultural implications of real-time music-making by people in disparate geographical locations. Even when not facing a global pandemic, telematic music is being widely explored for its potential to enable new modes of artistic collaboration, and in the process is suggesting entirely new art forms that are native to the network medium.

In the context of the COVID-19 pandemic, telematic performance is emerging as a necessity. The need for physical isolation and distancing has brought into relief the degree to which creative musical collaboration is reliant on sharing physical space. Musicians employ a variety of explicit and implicit gestural cues to coordinate their actions—a subtle nod of head, an exaggerated rise of the torso, or a fleeting glance can all signal vital information to co-performers. Our previous experiments, and those of others, have shown that this ability for visual coordination among performers, and visualization of remote performers for audiences, are among the most pressing needs in telematic performance.

The technologies underlying telematic performance currently allow us to achieve “CD quality” audio with sufficiently low latency between locations on the same continent to enable rhythmic music performance. Video, however, is a different story. The very process of encoding a digital video signal can be too slow to enable rhythmic coordination, not even factoring in transmission and decoding time. Even if we could push the limits further with video, there is evidence that video might not actually be the most effective way to support what musicians and audiences need. Research has shown that even at the same scale and information rate, physical movement in space is more engaging than video, and supports more effective interpretation of gestures. Video has limited ability to communicate complex human qualities such as “effort” and “tension” that are essential in music, but which involve extremely subtle and highly nuanced gestures or isometric muscle activations. Finally, video suffers from aesthetic deficiencies. In telematic performance, where we rely on the network to support metaphors of extending or sharing space, video only highlights spatial disjunction. We will always see artifacts of the “other” environment; we’ll never make it “look” like we’re sharing space, even if we can make it sound like it.

The goal of this project is therefore to explore methods of incorporating visual communication of effort, gesture, and movement into telematic performance without video transmission. We will conduct a series of practical experiments with different sensing techniques in isolation and combination, including infrared motion capture, inertial measurement, electromyography, and force sensing. These will be coupled with novel digitally fabricated mechatronic displays—simple moving avatars or kinetic objects that display the actions and efforts of remote musicians in 3-dimensional, physical space.

The goals of the UARTS Faculty Engineering/Arts Student Team (FEAST) will be to:

• experiment with and evaluate different sensing techniques for capturing performers’ movements and effort;
• design, fabricate, and evaluate at least one ‘finished’ mechatronic display for communicating musicians’ movement and effort in performance;
• hold at least one public concert to exhibit the technology, test it in a high-stakes environment, and gather qualitative feedback from audiences and performers to inform future research.

There is clearly no singular design outcome that would ‘solve’ the research problem, so future teams will likely iterate and improve on the designs that are generated as well as generate new ones. The designs will also likely be specific to different instruments and even individual performance styles, so we anticipate expanding to different instruments, musicians, and musical contexts. Assuming these practice-based explorations are fruitful, we also envision conducting formal experiments with both musicians and spectators that will compare the effectiveness of audio, video, and mechatronic displays. The results of those experiments in turn will inform future design phases.

Meeting Details
TBD
Modality: In-person (interested in the project but unable to be on campus? Contact us to inquire!)