Gestural Control of Music
Using the Vicon 8 Motion Capture System

This page contains video demonstrations of mapping motion capture data to sound/music, using the MCM (Motion Capture Music) software. The examples were made using the Macintosh version, MCMMax. In these examples we deliberately use rather simple and direct mappings, in order to make the relationship between gesture and sound as evident as possible. While not musically rewarding in their own right, the examples demonstrate some of the possible strategies for dancer-controlled music.

The sound in all of these examples was generated by translating and mapping motion capture data into music control information in real time using MCMMax. Thus, all of the sound was produced directly by the dancer's movements, not by an instumental performer. The intention is to demonstrate the generation and control of the music by the dancer, as opposed to a dancer moving to a musical accompaniment produced by someone else. While the mapping of motion to music happens in real time, it should be noted that these examples were not produced by a dancer who was hearing the music s/he generates (as would be possible with a fully realtime system); instead, these examples were made using pre-recorded motion, and that motion was played back to produce the sound in real time. As a next step, we hope to perform experiments with a fully realtime (Vicon RT) system, which will allow us to explore the interactive relationship between a dancer and the music s/he generates.

Example 1

This first example demonstrates tracking the location of a single point on the body - in this case the dancer's right shoulder. To make this tracking as explicit as possible, the x,y,z coordinates of the shoulder are used to control a single oscillator, somewhat in the manner of a Theremin. The vertical location (height) of the shoulder controls the pitch of the oscillator. The proximity of the shoulder (to you the viewer) controls the loudness of the oscillator. The left-to-right position of the shoulder in the performing space (from the point of view of you, the viewer) controls the left-to-right panning of the sound between the stereo speakers.

Example 1b

This is a variation on the first example, but tracking the location of both shoulders. The mapping of location to sonic parameters is exactly as in Example 1, but with one oscillator assigned to each shoulder, such that the two "Theremins" perform a closely related counterpoint.

Example 2

This example shows the same motion as the previous examples, and tracks the same point on the body, but maps the coordinates to discrete MIDI notes and other MIDI control data. The height of the right shoulder again is mapped to pitch, but rather than a direct mapping of height to pitches in the chromatic scale, the actual MIDI pitches are obtained from a lookup table containing an arpeggiated chord with blues inflections. Thus, a (very!) minimal knowledge base of musical culture is accessed by the dancer's shoulder. The volume and panning (MIDI controllers 7 and 10) are mapped to the shoulder in the same manner as the previous examples. (Note: This example movie has no audio track. It has instead a MIDI track containing note, volume, and panning information. You will need to have the QuickTime musical instruments or comparable software synthesizer installed on your computer to hear this example.)

Example 3

Rather than using location to change the pitch, we can use the location of a point on the body to control the timbre of a sound. In this example, we use the height of the right shoulder to control the modulation index of a simple FM synthesis sound. This effectively controls the "brightness" or spectrum of the sound. We use the height of the left shoulder to control the harmonicity of the frequency modulation, varying between 0.971533 and 1.029303; this causes the note to go "out of tune" to varying extent as the left shoulder moves up and down. The loudness and panning of the sound are controlled in the same manner as Example 1. We use only a single fundamental pitch for this example, to make the timbre modifications more evident, but the same principle could be applied to synthesis of melodies and chords (with the pitches and rhythms being performed by an instrumentalist or by tracking other points on the dancer's body).

Example 4

This example uses the same type of frequency modulation as in the previous example, but for controlling the modulation index, instead of tracking the location of a single point on the body, we track the distance between the tips of the left and right feet. Other parameters are the same as the previous example. By using relative distance between two points on the body, rather than location in space of a single point, control of the parameter does not depend on the dancer's location or orientation in the space. It also shows how the mapping can correspond to a natural periodicity in a person's movement, in this case the way the distance between the feet varies when one walks.

Example 4b

This is similar to the previous example, but instead of tracking the distance between the feet, we are tracking the distance between the finger of the right hand and the right side of the forehead. Since the position of the head on the body does not vary greatly (fortunately), it is easy to predict the effect of a given gesture by the right hand. In this example, as in the previous FM examples, the modulation index varies from 0 to 20; here the variation is in direct proportion to the proximity of the right hand to the head, which permits very specific gestural control of the parameter.

Example 5

Rather than generating music directly, the dancer can modify music being played from another source. In this example the locations of points on the body are used to control parameters of a lowpass filter applied to a loop of pre-recorded music. The left-to-right location in the space of the dancer's right shoulder controls the cutoff frequency of the lowpass filter, varying between 100 Hz and 4000 Hz, and the height of that shoulder controls the filter Q (the steepness of the filter and the resonance at the cutoff frequency). Additionally, the height of the left shoulder controls the playback speed (and thus the pitch) of the sound file, varying between 0.8909 and 1.22462 of the original speed (up or down as much as a whole tone in pitch), and the height of the right hand controls the amplitude.

Example 6

Similarly to example 5, in this example the dancer controls a filter altering a pre-recorded string piece. The relative distances "left hand-pelvis" and "right hand-pelvis" are linearly mapped to the modulation index and amplitude of a filter shaped by frequency modulation.

Example 7

Instead of controlling continuous musical parameters, as shown in the previous examples, the trajectories can also be segmented in order to control discrete musical events. In this example, the segmentation is set by determining maximum or minimum values of the relative distances "pelvis-head", "pelvis-shoulders", "pelvis-elbows", "pelvis-hands", "pelvis-knees", and "pelvis-feet". Each of these points triggers various sound samples, chosen either deterministically or randomly.