Add or Delete a Painter’s Style Using Neural Algorithms
London-based new media artist Kyle McDonald tells The Creators Project how he hijacked "Deep Dream"-style networks to transfer iconic styles between artworks.
A few weeks back, London-based new media artist Kyle McDonald got interested in a paper called “A Neural Algorithm of Artistic Style.” Writing on Medium, McDonald talks about the process, calling the results “some of the most compelling imagery generated by deep convolutional neural networks (DCNNs) since Google Research’s ‘DeepDream’ post”.Basically, the algorithm overlays a famous painter’s style—think Vincent van Gogh’s swirling textures—on top of a photograph. Researchers ran a photograph of riverside buildings through the neural algorithm, and were able to get it looking like Picasso, Edvard Munch and Kandinsky paintings, amongst others.
Advertisement
McDonald wasn’t really interested in adding artistic styles to an image, however, but in removing an artist’s style from a painting. McDonald attempted it with Vincent van Gogh’s “Wheat Field Behind Saint-Paul” and “Green Wheat Field with Cypress,” but it didn’t exactly work—the paintings looked digital and blurred, but the overall style wasn’t obliterated.As a joke on his failure, McDonald remixed a photograph taken by Tadeo Cern for “Revealing the Truth”—featuring a van Gogh lookalike—by running it through Photoshop and adding some glitch. McDonald then posted it to Twitter with the text “gogh, un-goghed.” Again, it was a joke, but some of his followers—The Creators Project included—thought he’d succeeded.
McDonald let us know that he hadn’t, but did tell us how he attempted to hack the neural algorithm. The paper’s authors say that the algorithm captures “style”, but McDonald says it actually captures “texture”—the thing that doesn’t change across the entire painting.“My first thought was to strip paintings of their style, and basically hallucinate the original scene from a heavily stylized image,” McDonald says. “This may be possible in some small set of situations, but most of the time it just results in some weird mashup of a painting and a photo.”
When McDonald applied the algorithm to van Gogh’s “Wheat Field Behind Saint-Paul” remix, he became somewhat fixated on what happened to a minor part of the artist’s original.“I was most drawn to a small barn in the distance that was turned into a lump of dirt, presumably because that's was the best color match,” McDonald says. “It made me wonder what Van Gogh saw, if maybe there was a lump of dirt that he turned into a barn for us.”
Picasso x Porn from Kyle McDonald on Vimeo.McDonald says there very few paintings that can be “un-styled” in this way. Mainly some post-Impressionism, perhaps some Pointilism. He says that those artists have embedded a ton of information into their paintings, from how color gradients flow across to the image to the points where there are more or less texture, and so on.“It's totally removed from realism while still having all the same information as a photo,” he says. “Conceptually, that it still ultimately fails to produce a clearer image than the original work, this says to me that we might see things in a very different way than neural networks.”
McDonald did have success applying the algorithm in the creation of GIFs and short videos. In short order, McDonald posted two videos on Twitter: “walking through an abstract landscape” and “starry night in nyc.” The first is simply some feet hitting the pavement rendered in a highly abstract expressionist style, while the latter is an aerial shot of New York City fashioned to look like Van Gogh’s “Starry Night” painting.
Advertisement
Andy Warhol’s photograph mashed up with Ernst Haeckel’s style, courtesy of Kyle McDonald
McDonald also tested some “style fading,” an animation that starts with one style and finishes with another. The effect—again, the riverside building image is run through the algorithm, this time with animated effects—is subtle but interesting.“The process for making videos is almost the same as the stills: first I take a single frame of video and test style transfer with a few different parameters, like whether to use white noise, how many iterations to run, what scale the texture has, which style to borrow, etc.,” McDonald explains. “When I find something that feels right, I let it run through all the frames and combine it into a video or GIF. I followed this process to create the ‘abstract landscape’ with an early version of some code by Kai Sheng Tai, then ‘starry night in nyc’ with the code from Justin Johnson, as well as ‘Picasso x Porn.’”At the same time, McDonald spent some time digging through Western art history from cave paintings to Minimalism and Pop Art, looking for unique and recognizable styles. He then mixed them up with a few photos used in iconic series: Marilyn Monroe as Andy Warhol’s subject and Mt. Fuji as Hokosai’s subject. McDonald posted these experiments on his website under the title “Style Transfer Studies,” using code written by Justin Johnson based on the original research paper.
A photographic portrait blended with painter Gustav Klimt’s distinctive style.
“Some of them capture the likeness of the original work perfectly, others would need to be tweaked to make more visual sense,” McDonald says. “The mosaics from Roman and Byzantine art work well: it seems to understand the overall flow of the image locally as well as the textures. It seems to understand dot stippling, watercolor gradients, large color regions.”
Advertisement
McDonald says it fails miserably at recreating Rembrandt’s style, painting the lighter colors as a foreground and turning everything else black. As he emphasizes, what the algorithm does is about texture rather than style.
A photographic portrait blended with ancient mosaic technique.
ONE EMAIL. ONE STORY. EVERY WEEK. SIGN UP FOR THE VICE NEWSLETTER.
By signing up, you agree to the Terms of Use and Privacy Policy & to receive electronic communications from Vice Media Group, which may include marketing promotions, advertisements and sponsored content.
McDonald wasn’t really interested in adding artistic styles to an image, however, but in removing an artist’s style from a painting. McDonald attempted it with Vincent van Gogh’s “Wheat Field Behind Saint-Paul” and “Green Wheat Field with Cypress,” but it didn’t exactly work—the paintings looked digital and blurred, but the overall style wasn’t obliterated.As a joke on his failure, McDonald remixed a photograph taken by Tadeo Cern for “Revealing the Truth”—featuring a van Gogh lookalike—by running it through Photoshop and adding some glitch. McDonald then posted it to Twitter with the text “gogh, un-goghed.” Again, it was a joke, but some of his followers—The Creators Project included—thought he’d succeeded.When McDonald applied the algorithm to van Gogh’s “Wheat Field Behind Saint-Paul” remix, he became somewhat fixated on what happened to a minor part of the artist’s original.“I was most drawn to a small barn in the distance that was turned into a lump of dirt, presumably because that's was the best color match,” McDonald says. “It made me wonder what Van Gogh saw, if maybe there was a lump of dirt that he turned into a barn for us.”Picasso x Porn from Kyle McDonald on Vimeo.McDonald says there very few paintings that can be “un-styled” in this way. Mainly some post-Impressionism, perhaps some Pointilism. He says that those artists have embedded a ton of information into their paintings, from how color gradients flow across to the image to the points where there are more or less texture, and so on.“It's totally removed from realism while still having all the same information as a photo,” he says. “Conceptually, that it still ultimately fails to produce a clearer image than the original work, this says to me that we might see things in a very different way than neural networks.”
