A version of this article originally appeared on The Creators Project Germany.
This article was originally published on November 25, 2014 but we think it still rocks!Berlin-based interaction design studio Schnellebuntebilder are masters when it comes to transforming physical movements into stunning visuals. With MOMENTUM, they've teamed up with sound designers Kling Klang Klong to create "a synaesthetic experiences of sounds and visuals generated real-time from body movement," according to Schnellebuntebilder's Magdalena Eder.
Advertisement
MOMENTUM transforms the user into a sea of digital particles. It consists of "several blocks with different functions called nodes," explains Schnellebuntebilder's John Timpernagel. "There is a node that controls the Kinect2, created from the RGB data and infrared image of the dancer. From that data, [MOMENTUM] creates point-clouds that capture the depth of various points in space.""We connect the data to a graphical system which puts the dancers' particles in motion. A hand movement to the right can push them [the particles] to the right," says Timpernagel.Originally, MOMENTUM could only link movements to sounds and sound dynamics, which came from Kling Klang Klong. Based on the readouts of data points, paramaters were mapped to dancers' entire bodies. Quick hand movements could trigger wind noise. Making one's self small by squatting on the floor could induce a low-cut filter.The soundscape consists of several layers. The first, a pre-produced track, forms MOMENTUM's musical backbone upon which the other layers are generated in real time. Parameters such as pitch can change over the course of the dancer's interactions.
Based on the generated sounds, the body's movements are also transformed into colorful images. "We had the data already, so now we could make nice visuals. MOMENTUM's first graphical interface was relatively ugly—now, we've connected the particle system to the data," Timpernagel tells The Creators Project.
Advertisement
The results look like flowing, colorful virtual reality creatures who exist in 3D space. It's a synesthetic experience of liquid forms and sounds that allow users to control and manipulate dynamic models with their own bodies.
ONE EMAIL. ONE STORY. EVERY WEEK. SIGN UP FOR THE VICE NEWSLETTER.
By signing up, you agree to the Terms of Use and Privacy Policy & to receive electronic communications from Vice Media Group, which may include marketing promotions, advertisements and sponsored content.
