Learning Linear Transformations for Fast Arbitrary Style Transfer
https://arxiv.org/abs/1808.04537
Not published but looks like CVPR submission, 2018.

❓ What?
The paper is a follow-up of the WCT ("Universal Style Transfer via Feature Transforms", Yang et al., NIPS 2017).
The basic idea is to transfer second order statistics from a style image onto a content image via a multiplication between content image features and a transformation matrix. In the WCT paper authors calculated the transformation matrix with a pre-determined algorithm. In this paper, they propose to use a neural network to produce the desired transformation matrix based on a pair of style and content images.

✏ Short algorithm explanation:
1. Train VGG-based autoencoder on the content images' dataset.
2. Add a transformation module in the bottleneck, which will take content image features and style image features and produce a transformation matrix.
3. Apply the transformation to the content image features, feed the transformed features to the decoder and get a stylization.
Steps 2 and 3 are trained with widely used Gram matrix losses on pretrained VGG features (Gatys et al., 2015).

📢 What is the claimed benefit?
1. Using several different layers of VGG network to compute Gram matrix loss is the commonly adopted technique in style transfer.
In case of WCT, one would have to do k forward passes through the stylization network to use k different layers of VGG.
On contrary, the proposed method will learn to model statistics of those k layers in a single transformation matrix, which is more efficient.
2. After training a stylization can be produced with any previously unseen style image in a single forward pass.
3. The learned transformation allows the usage of a shallower encoder.
4. It's is fast- 40 FPS on 512x512 images with shallow encoder and 28 FPS with a deeper one.
5. More stable video stylization in a frame-wise manner, w/o temporal context.
4. According to the provided figures, they improved stylization quality compared to WCT and AdaIn (Huang et al., ICCV 2017).

✔ Experiments:
- Standard style transfer experiments, but not many methods which they compared to.
- Video stylization.
- Photo-realistic stylization (like day to night).
- Game to real (GTA images to photos).

⛔ Criticism:
- Poor comparisons to the existing methods. I have a strong impression that images are cherry-picked to show the cases where improvement is visible.
- Doubtful possibility to apply the method to images larger than 512x512 pix.

🔚 Take home:
Matching second order statistics between content image and style image can be modeled by a linear transformation, which is generated by the module. This gives a fast and generalizable approach for style transfer.

🔻 Links:
[1] WCT https://arxiv.org/abs/1705.08086
[2] AdaIn https://arxiv.org/abs/1703.06868
[3] Gatys et al., 2015, https://arxiv.org/abs/1508.06576