I Used Leap Motion to Touch a Virtual Rainbow Universe

Love and rejection were important themes at this year's Indiecade East, which fittingly took place on Valentine's Day weekend at Museum of the Moving Image. I tried out a kissing simulator, an arcade-style game that awards points based on how many times you can make one liberated Norwegian woman get lucky, and a mutliplayer game that featured four narwhals trying to pierce each other's hearts, in addition to titles from Nintendo, Sony, and Microsoft. But I had the most fun in the Leap Motion corner, where the motion-tracking hardware company had six different virtual reality experiences set up. I swiped my way through a star map the size of the night sky, explored a virtual space station, and traveled through time on a sci-fi rescue mission, controlling each experience with just my bare hands. One game stood out, though: a synesthetic world that I believed I could touch and feel, although I was only grasping thin air.

I slid the Oculus Rift goggles over my eyes and immediately faced a giant, bearded head, seemingly made of an oil slick. This was the debut of Rainbow Membrane, an experimental audiovisual video game designed by Leap Motion developer Isaac Cohen to fool my hands and brain into believing they could actually feel the virtual walls, floor, and giant face that surrounded me. As Cohen explained, "By creating a physical interaction in a digital space, Rainbow Membrane attempts to break down the false dichotomy between the 'virtual' world and 'physical.'"

Inside Rainbow Membrane, it was easy to gaze at the shimmering walls and admire the sculpted chin of a figure Cohen described as "not necessarily Abraham Lincoln," but the real magic began once I stretched my hands out in front of my face. I realized I could not only touch but feel everything I was seeing, thanks to a combination of the surface's realistic physics and the sound cues I was hearing. Cohen explained how the synesthetic environment worked, stating, "Audio is placed around the user, so as they caress the skin-like film, they also create a song that is unique to their movements through the field." It seemed like a simple enough concept; foley artists have been fooling the eyes via the ears since The Jazz Singer. But the idea that vision and sound can fool the fingers into feeling is something that has to be felt to be believed.

The basic seed of Rainbow Membrane sprouted when Cohen and friends decided to design a Justin Bieber-shaped punching bag. The developer began work on a moldable digital mesh that would simulate the way a face looks in slow motion when it gets punched, but the project fizzled out when he couldn't find a believable-enough 3D model of the pop star. He was, however, left with an ancestor of the silky digital skin that now covers Rainbow Membrane. With help from 3D designer Steven Teeps, he built the colorful model and coded it in three.js, embedding a musical loop for each white dot that hangs inside the model. It was here that game's synesthetic effects emerged: as I dragged my hands across responsive walls and floors, the music got louder, tricking my brain into "feeling" the virtual space around me.

Later, Cohen explained that everyone he dropped into Rainbow Membrane had a different way of describing the walls' texture, including, "felt, velvet, a waterbed, skin, a furry wall, water with algae on the surface, and my personal favorite, the thin skin of burnt milk on the top of a cup of hot cocoa." But despite their various descriptions, each account actually only described the feeling of thin air, as filtered through Cohen's illusion.

While he has no plans to expand Rainbow Membrane, Cohen has a lot of ideas about the future of virtual reality. He's used skin-like textures in projects that ranged from a series of touchable floating orbs, to an "album / gallery / synesthetic wonderland" called DRAGONFISH, and is currently at work on an immersive children's book and a suite of intensely emotional experiences, including one where you get to lie down and play with a cosmic mobile (including a puppy!) that floats above you.

"I think that most people like talking about things like Minority Report or Iron Man-type interfaces when they talk about the future of VR and motion control, but I'm so much more interested in the capabilities of these technologies to create visceral, emotional experiences," Cohen explained. "Rather than thinking about how they will make us more efficient at accomplishing tasks, I wonder more about how they can positively affect our humanity." In his experience, the best games get players exploring and questioning, rather than "numbing or distracting us from our miraculous human forms." Cohen himself uses Rainbow Membrane to meditate and relax after a stressful day in a way that doesn't shut his mind off, and a huge part of Leap Motion's promise lies in a future stimulated, literally hands-on gaming.

Check out more of Cohen's work on his website.

Related:

Zero-G Simulator Lets You Float Through an Oculus Rift Space Station

I Ate Clouds At The World Maker Faire

Walk, Run, and Fly Through Vincent Morisset's New Interactive Film

A Virtual Reality Experience Gave Me Synesthesia