The Environment Is Your Playground in an Interactive 3D Ecosystem

The Great Hall in the New York Hall of Science has undergone a transformation: The curved walls of the iconic space, originally built for the 1964/65 New York World’s Fair, are currently home to six interactive ecosystems forming a vast. interconnected digital environment of fantastical virtual organisms. It forms an exhibition called Connected Worlds which opened on the 27th June and was designed and created by Design I/O—artists Theo WatsonEmily Gobeille, and Nick Hardeman

The huge piece—the largest interactive installation using projection ever created, the artists say—encourages visitors to engage with the six ecosystems contained within the space using gestures and movements. All the walls, including the 2,300' floor space, is interactive and it also contains a 38' virtual waterfall which feeds into the environments.

As visitors interact with the virtual plant life and animated animals in one area, their actions will lead to consequences in another—with all interactions based on principles of sustainability science. The aim is to help children and visitors understand the interconnectivity of the natural world and demonstrate, in a hands-on manner, the importance of balance in maintaining a healthy ecosystem.


Connected Worlds. Photos courtesy of Design I/O

Bringing such a huge exhibition to fruition has been a long process and has been brought to life using 12 Kinects, infrared tracking and numerous 20,000 lumens projectors. To find out more about this incredible and complex digital biosphere. we asked Theo Watson a few questions over email.

The Creators Project: How did you come up with the designs for the characters and landscapes? Were they all based on real creatures/places?

Theo Watson: The landscapes are based on real biomes (Desert, Mountain Valley, Plains, Jungle, Reservoir, and Wetlands), but we wanted the creatures and plants to be more fantastical and less real so that the visitors would be forced to look at them from an unbiased viewpoint and try and discern what their appearance and behavior signified. A lot of the creatures are dependent on having a certain plant thriving in the environment. For example, the Desert Stoic Bird appears in the Desert when the Shifty Cactus is planted there and is healthy, but it will also migrate through the Mountain Valley to the Plains when the Beacon plants are pulsing.


Designs for the plants. Photos courtesy of Design I/O

You have kinects, tracking, projectors, there's lots going on. Can you describe a little how the setup works? 

The project uses a total of 15 projectors. The floor is made up of seven projectors stitched together using Pandoras Box blending hardware. The wall environments are made up of six 20,000 lumens projectors and the waterfall is two 20,000 lumens projectors mounted in portrait mode. The whole installation runs on eight Mac Pros and the software is built on top of openFrameworks: an open source, creative coding toolkit for C++.

For the water routing on the floor, visitors can use physical logs to setup paths for the water to flow along. Anytime the water hits a log it will flow along it, so visitors can create quite complex paths (which often need adjusting as changes happen upstream). We are tracking the logs with three IR cameras and IR lights from above. The logs are made of retroreflective fabric which allows us to ’see’ the logs easily across the space as it reflects 95% of the light from the IR lights back to the cameras. 


Connected Worlds. Photos courtesy of Design I/O

And how does the interactivity work?

For the interactivity in the environments along the wall, children can make seeds with their hands by holding them out in front of them, they can also cut down dead trees by making a chopping gesture. In some environments visitors can trigger behaviors in the plants, like releasing water from the Shower Plant in the Wetlands. Visitors can also interact with the creatures, for example chasing the Rock Roller creatures out of the Mountain Valley stream where they block the flow of water to the floor. For the wall interaction we are doing custom person tracking and hand detection from above using two Kinects per environment. We combine the point clouds from the two Kinects together into a single virtual space which allows us to get rid of the perspective change as you move from one Kinect to the other Kinect.

There is a bit more of a thorough explanation of the floor and wall tracking in this behind the scenes video.

The different ecosystems interact with each, with causes and effects and characters migrating based on these, etc. How did you go about doing the programming for this? Is it all based on how these ecosystems work in the natural world? 

We draw inspiration from the natural world for a lot of the interactions between the environments, but we try to make it clear that while Connected Worlds is analogous and references the natural world, it is a fictional ecosystem with its own unique behavior. Because it is non-real and a little mysterious, this encourages visitors to look at it as a puzzle and to try and piece together the different things they are seeing and reveal the rules that govern this unique environment.  


Connected Worlds. Photos courtesy of Design I/O

What were some of the challenges in bringing a concept like systems thinking to an installation for children?

Systems thinking seems like a fairly challenging concept to teach but its fundementally quite intuitive to grasp as we are constantly interacting with systems in our daily life. I think the challenge for Connected Worlds was how to make sure the systems thinking component of it didn’t get lost underneath the overall complexity of having a series of very visual and dynamic interactive environments. 

Originally we had thought to have more complex rules governing the populations of creatures (i.e: invasive species, or population explosions) but we realized that for the systems thinking component to really come through, we couldn't present multiple parallel systems but instead focus on one (water/resource management) so that it becomes the single defining factor of how well the environments are doing.


Connected Worlds. Photos courtesy of Design I/O

And what were some of the challenges in creating such a large connected and complex digital eco-system, while also making it simple enough for children to engage with?

We realized that the individual interactions could be fairly simple (planting seeds, chopping dead trees and routing water, as when you combine six environments together all sharing a common water source and there is a finite amount of water in the system, this would create a lot of complexity. There are also time delays which play a factor (how fast the clouds return water to the waterfall, etc.), so we knew we shouldn’t add too much before play testing the entire system. As it turns it out is really quite hard, even with the fairly simple rules.

Getting water to all of the environments is fairly challenging when you are doing it over a 68' space. Keeping the interactions simple makes it easy for children to jump into the project, the complexity really comes from the decisions they individually and collective take and how those decisions impact the system as a whole, which in turn feeds back and affects what they are trying to achieve. 


Connected Worlds. Photos courtesy of Design I/O

Do children react very naturalistically to these kinds of immersive virtual worlds do you find? And do you also find it genuinely facilitates and aids their learning about the concepts like interconnectivity and the natural world?

As part of the development of Connected Worlds we have run a bunch of play testing sessions and with partners from NYU conducted interviews and formal evaluations with groups of children. We were actually quite surprised how many of them could explain (in detail) how the system was working and sometimes picking up really subtle behaviors, as well as connecting it to what they saw the ‘bigger message’ to be.

It was amazing to listen to children of a wide range of ages make really poignant observations and also connect it to concepts they had been discussing in class. I really hope we’ll be able to publish some of this research as it was really informative to us, helping shape the final concept and also giving us the confidence that we were on the right path with our approach to Connected Worlds


Connected Worlds. Photos courtesy of Design I/O

Learn more about this and other Design I/O's projects on their website.

Related:

Artist Captures an Ocean Storm in a Fishbowl

Now You Can Experience Miami's Neon Coral Reefs with Oculus Rift

Someone's Been Projecting Crying Babies onto Smog