Image Style Transfer (CNN)

Summary

This paper discusses a "Neural Algorithm of Artistic Style" that can disentangle and blend the image content and style of natural images." While previous algorithms could handle texture transfer, they were mainly effective for low-level image features of the target image.

This paper utilizes Convolutional Neural Networks (CNNs) trained with ample labeled data for tasks like object recognition. These networks extract high-level image content in generic feature representations that are applicable across datasets and even to other visual information processing tasks, including texture recognition and artistic style classification.

This research applies the feature space given by a normalized version of the 16 convolutional and 5 pooling layers of the 19-layer VGG network.

Style Transfer: Goal of formulation

Goal : $\mathcal{L}_{total}(p^ ⃗,a^ ⃗, x^ ⃗) = \alpha L_{content} + \beta L_{style}(a^{ ⃗}, x^{ ⃗})$

Content representation

$\mathcal{L}_{content}(p^ ⃗, x^ ⃗,l) =\frac{1}{2}\displaystyle∑_{i,j}(F_{ij}^l-P_{ij}^l)^2$

Here, $p^ ⃗$ represents the original image, and $x^ ⃗$ represents the generated image. The content loss between the two features can be computed by the provided formula.

By training, the random image $x^ ⃗$ can match the feature of the original image $p^ ⃗$ in a specific layer of the CNN. As we go higher in the layers, the network captures high-level content, focusing on the overall content rather than the exact pixel values of the input image. This is defined as content representation.

Style representation

Correlations between various feature maps are used to get the style of the input image. This correlation can be represented by the Gram matrix. The texture information of the input image can be obtained using the feature correlation of several layers.

Here, $a^ ⃗$ represents the original image, and $x^ ⃗$ represents the generated image. We can define loss by reducing the distance between the original image and the Gram matrix of the image to be generated as described in the paper.

Total style loss is the sum of the losses of each layer weighted by the weighting factor $W_l$.

$p^ ⃗ : original \ image$

$x^ ⃗ : generated \ image$

If the features of $p^ ⃗$ and $x^ ⃗$ encoded in layer are $F_l$ and $P_l$ , respectively, the content loss between the two features can be defined as follows.

$\mathcal{L}_{content}(p^ ⃗, x^ ⃗,l) =\frac{1}{2}\displaystyle∑_{i,j}(F_{ij}^l-P_{ij}^l)^2$

As you learn by changing random image $x^ ⃗$ using back-propagation as follows, it becomes the same as the feature of the original image $p^ ⃗$ in a specific layer of CNN.

$\frac{\delta \mathcal{L}_{content}}{\delta F^l_{ij}} = \begin {cases} (F^l-P^l)_{ij} & \text{if} \ F_{ij}^l > 0 \\ 0 & \text{if} \ F_{ij}^l < 0 \end {cases}$

In addition, as you go back to the layer, the network captures the high-level content, so you focus on the overall content rather than the exact pixel values of the input image. Accordingly, the feature response of the higher layer was defined as content presentation.

Style transfer

An image with white noise is inputted, to transfer the artwork $a^ ⃗$ (style image) style to the photograph $p^ ⃗$ (content image). The total loss function is defined as:

$\mathcal{L}_{content}(p^ ⃗, x^ ⃗,l) =\frac{1}{2}\displaystyle∑{i,j}(F_{ij}^l-P_{ij}^l)^2$

Correlations between several feature maps are used to obtain the style of input image. This feature correlation can be expressed in the following Gram matrix.

${G}_{ij}^l =\displaystyle∑_{k}F_{ik}^lF_{jk}^l$

Texture information of input image can be obtained using feature correlation of several layers.

$a^ ⃗ : original \ image$

$x^ ⃗ : generated \ image$

If the style presentations of $a^ ⃗$ and $x^ ⃗$ in the layer are $A_l$ and $G_l$ , respectively, loss can be defined by reducing the distance between the original image and the Gram matrix of the image to be generated as follows.

${E}_{l} =\frac{1}{4N_l^2M_l^2}\displaystyle∑_{k}(G_{ij}^lA_{ij}^l)^2$

If the content of each layer is defined as weighting factor $W_l$, total style loss is as follows.

$\mathcal{L}_{style}(a^ ⃗, x^ ⃗) =\displaystyle∑_{l=0}^LW_lE_l$

Overall

The paper describes the learning process of the Style transfer algorithm as such:

• Initially, a random noise image was entered in the image to be created.

• The content Image calculates and propagates only the loss of the high-level layer.

• The style image calculates the loss for each layer and propagates it over.

• Total Loads is the weighted sum of the two loads.

Code (Tensorflow 2.0)

GitHub - erickim20/tf-style-transfer