Neural Styles and Bioshock's Concepts
Part 1: Adapting neural style transfer for creatives.
In the early days of computer graphics, you saw a lot of uninspired crap like this.
The tools were in the hands of computer graphics engineers, not yet in the hands of artists and designers.
Now, except for a handful of Luddites, familiarity with software tools for creativity is de rigueur for artists and designers; no art or design student can expect to be taken seriously if not familiar with Photoshop, Illustrator, Procreate, and similar tools.
Looking forward, I wonder if and how AI-based tools will enhance creative workflows as software did.
Neural style transfer and Bioshock
Neural style transfer is a deep learning technique where the style from one image is applied to the content of another image.
It is certainly fun. I began wondering if neural style transfer is useful to a designer or an artist, and if so how.
To this end, I started browsing the gallery of images made by people on DeepArt.io, a neural style transfer app. There are several genuinely impressive results in the gallery. These are a few of my favorites.
Then I noticed this one.
DeepArt.io showcases this picture in its gallery, but I think it was a failure. I recognize the character in the bottom left. She is Elizabeth Comstock from the Bioshock video game series. The style’s source image on the bottom right is work by Alphonse Mucha, perhaps the most well-known artist of the Art Nouveau style.
Know someone who might think this is interesting?
I doubt the content and style choice is a coincidence because the Bioshock series explicitly uses Art Nouveau (as well as Art Deco) in its concept art. I suspect whoever tried this bit of neural style transfer wanted a depiction of Elizabeth Comstock in a canonical Art Nouveau theatrical poster style. Something like this work by illustrator Bill Mudron.
So neural style transfer failed. This person wanted to apply an Art Nouveau theatrical poster style to the character the way you apply a font to typed text in a word processor. They did not want abstract art.
The problem comes down to how we define the “style” in “neural style transfer.” In neural style transfer, “style” means statistically regular hierarchies of colors, geometries, and textures. Colors, geometries, and texture can surely pack an emotional punch. But clearly, that which makes a Mucha a Mucha is more than just geometry, shapes, and colors.
To illustrate, let’s look at a successful neural style transfer on the Elizabeth Comstock character. In one of the sequels in the Bioshock series, Elizabeth returns with a new femme fatale style.
This main difference between this version of the character and the other is the color scheme. Neural style transfer can easily handle transfer of color.
However, things become difficult as the elements of the style we wish to transfer become more complicated.
For example, the femme fatale Elizabeth appears in a dystopic deep-sea city that borrows aesthetic style elements from the 1933 Chicago World’s Fair (left). The original Elizabeth appears in an equally dystopic floating city (right) with aesthetic elements borrowed from the 1893 Chicago World's Fair.
Here, one might ask, “What might the cloud dystopia look like with the deep-sea dystopia’s aesthetic?”
I call this a fail. Success would have seen the architecture of those flying buildings adopt the Art Deco skyscraper architecture of the style source image, and if the buildings looked as though they were under water rather than just bathed in blue.
Towards concept transfer
The problem is that neural style transfer fails when the style in question is defined in terms of high-level concepts. Neural style transfer transfers colors, geometries, and shapes. Visual concepts are composed of these elements, but also real-world ideas like gravity and schools of fish and how light refracts through water. These ideas are the irreducible primitives that artists and designers reason about when creating.
Artists and designers deal in concepts. Concepts are paramount, shapes and colors are how they render those concepts. To make such tools useful for creatives, we need concept transfer.
In next week’s post, I present a concrete “concept transfer” use case.
Please leave a comment if you have thoughts or find this interesting.