Paint Transformer: Feed Forward Neural Painting with Stroke Prediction

Earlier I wrote about a style transfer that, instead of optimizing pixels, optimizes directly parameterized brush strokes. A new work has been released, it uses Transformer architecture to predict stroke parameters using one (well, almost one) forward-pass of the network. In fact, their transformer is made in the spirit of a recurrent network.

Four forward passes of the original image in different resolutions are made through the network, starting from downsapled 16 times to the original one. At each next forward puss, a rendered canvas with strokes predicted in the previous passes is also added to the input as well as the original image in a higher resolution. Thus, the network gradually adds new strokes to the canvas, starting with larger ones (they are painted on a low resolution canvas), and ending with smaller ones (on a high resolution canvas). The network is trained on synthetic data generated online.

📝 Paper
🛠 Code