Behind The Scenes Of The CGI Times Square In 'The Amazing Spider-Man 2'

It’s nighttime in New York City and an angry blue man is conducting the power of electricity to flip cars and send Spider-Man flying hundreds of feet. It's just another evening in the storied life of your friendly neighborhood title character (played by Andrew Garfield) in The Amazing Spider-Man 2— except this time, when Jamie Foxx’s Electro realizes his full powers and rage, he's broadcast in front of a massive Times Square audience, surrounded by flashing signs and LED screens.

How does one go about creating such an environment? Times Square is difficult enough to shoot on a cell phone camera, let alone one that shoots in HD, but to add CGI effects that bring one of the world's most bustling districts to its knees is a tactical VFX nightmare. We were curious about how the special effects team brought this chaotic sequence to life, so we talked to Jerome Chen, visual effects supervisor onThe Amazing Spider-Man 2:

The Creators Project: So what was it like to digitally recreate Times Square?

Jerome Chen: First, we do a script analysis which means that we read the action that happens in the script. Then we determine, “How much of this can we actually shoot at Times Square?” We approach the city; obviously they aren't going to let us damage much of it. They only gave us one night there, [so we shot] Jamie Foxx’s character Electro walking into Times Square and kind of looking around.

We broke down the action through a storyboard phase before using a process called tweening, which is a very rough animatic of the scene done on a computer with 3D models. That gives me an idea of how much of Times Square needs to be physically constructed and how much needs to be digitally constructed. You have to have enough physical construction to shoot some of the scenes. They can’t all be done on the computer.

So you built a set of Times Square?

Yes, but obviously Times Square is huge. The only part of Times Square that we physically built was a small section of Duffy Square, the triangular part where the TKTS Stadium is. The red little bleachers. So flat pavement, a little bit of that raised triangular section, and then the actual stands themselves. We built that almost to true scale. Maybe 90% of the actual size. A tiny bit smaller than the real one, but not by much. That was the section we could physically shoot on, so we could have the bystanders on the stand. The police cars could drive in and Jamie Foxx could act in that area.

We built only one actual storefront, which was by the Roxy Theater on the west side. We surrounded that with a huge wall of shipping containers stacked fifty feet high. The shipping containers were covered with vinyl billboard material that we painted green. That gives me a green screen wall surrounding the constructed area of Duffy Square. The green screen offers a guide in the compositing phase of our production; it signals the area that I need to replace with computer-generated extensions of Times Square. At the end of the day, 90% of Times Square has to be created by computer. That’s where we get into determining how to create Time Square. We built this practical part on an exterior stage on Long Island.

How do you construct a digital version of Times Square? You need to start with an accurate geometric model of it, in computer. A company out of Baltimore that does laser scanning, or Lidar scanning, of objects laser-scanned Times Square during Thanksgiving and Christmas of 2012. We got permission from the city to collect survey data and scan every surface and every architectural detail of Times Square. It took these guys about a week to do this. What that gives me is a three-dimensional model of Times Square once it’s all pieced together. It takes months to piece this all together. Once that’s finished, I get a reasonably accurate model of Times Square. It’s like the first step of building a model kit. You get all the plastic pieces and glue them together but it’s grey. There’s no color on them, or anything.

At the same time, we have a team doing an image acquisition of Times Square. They took thousands of photos of Times Square, details of storefronts, of walls, getting textures of bricks, the surface wall details. The grit, the grime. Thousands of photos for references for the digital artists who later take these images and apply them to the digital surfaces of the geometry. At the same time, we used high resolution, S65 cameras, and shot footage of all the video monitors on Times Square. Almost 90% of the light in Times Square comes from Jumbotron walls.

It was like a documentary process, trying to be faithful to Times Square as it is, rather than a Spider-Man-specific Times Square.

Absolutely. It’s such a recognizable landmark that we didn’t want to take liberty with it. One of the reasons we shot so much of it in New York City and the stage work in New York state was to make sure the movie felt like it takes place in Manhattan. Use as many iconic images of the city as possible to place it there. It was important we had a faithful reproduction of Times Square. The next step is then to create new shading technology for our software to render the images so that they’re realistic in terms of the light and the way that that video walls affect the atmosphere in Times Square. We created this unique technology to make sure the imagery was believable.

How much of the process involves unique technology that’s being created just for the movie? 

Almost all the software tools used by the artists were created specifically for us. We have an internal software department that writes tools. The extensions are pretty significant in terms of the background. We basically only had a few people [on set]. Only 30% of the people were real ones that we photographed. Everything else behind them was supplemental, computer-generated using a crowd system. We use motion capture of real people to have their real movements as they’re watching and milling and running about. The scale of the extension was so big that it had to be populated with cars. There’s only ten to twelve real police cars and some additional cars on the actual set. Everything else behind there is all computer-generated cars.

Image credits: ©2013 Sony Pictures Imagework

The Amazing Spider-Man 2 is currently in theatres nationwide. For more information, visit the official site.

Related:

An Inside Look At The VFX That Brought "The Grand Budapest Hotel" To Life

[Video] Damsels, Doppelgängers & Dostoyevsky: The Devious Devices Behind "The Double"

Meet The Real World Designers Behind The Fictional Video Games Of 'Her'