Olivia Wiles*, A. Sophia Koepke*, Andrew Zisserman
https://arxiv.org/abs/1807.10550
ECCV 2018
❓ Briefly
Authors proposed a model that can control a
source face given a driving face to produce a generated frame with the same identity as the source face but the pose and expression from the driving face. The model is trained in a self-supervised manner on a large collection of video data. They also show that the generation process can be driven by audio (person speaking) or pose codes w/o further network training. No 3D models used, the method works on 2D frames.✏️ Method:
The model consists of 2 networks: embedding net and driving net.
1. Given a source frame embedding net predicts a vector field which after applying to the source frame gives a so-called embedded face (which essentially is a frontalized face).
2. Given a
driving frame driving net predicts a vector field (i.e. x,y shift for every pixel of the input image). This vector field transforms pixels from an embedded face to produce the generated frame.Driving net has an encoder-decoder architecture (U-Net and pix2pix based). The latent space (i.e. bottleneck features, called the driving vector) of this network encodes pose/expression/zoom/other factors of variation.
Note: The trick is to avoid translating an input image to output image directly on the pixel level (as for example CycleGAN does), but to predict a vector field which will transform the input image to something new. This allows training w/o a discriminator and explicit pose/expression labels.
◼️ Training is done in two steps:
First step:
Authors sample two random frames from the same video (same person). One is used as the source frame, another as the
driving frame. In this case, the loss function will be a photometric L1 loss between the driving frame and the generated frame (since the source frame has the same identity as the driving frame).Second step:
The new loss function is introduced: identity loss, which compares frames using features of the pre-trained network (VGG-11) for person identification.
Authors sample a source frame
sA of identity A, and two driving frames dA, dR. dA is of identity A and dR a random identity. sA, dA, dR are used as training inputs. This gives two generated frames g_dA and g_dR respectively which should both be of identity A.For a pair
(dA, g_dA) photometric L1 loss + identity loss is used; for a pair (dA, g_dR) only identity loss is used.◼️ Inference step:
The pose and expression of the
source face can be changed by feeding to the model a driving frame with the face of any other person.Additionally, authors show that the transformation of the
source face can be done using a pose code (encodes face attributes such as pitch/yaw/roll angles, for which the ground truth is provided in the dataset). They train a 1-layer neural network to convert pose code into a driving vector (bottleneck features of the driving net) and use the driving net to generate a corresponding driving vector field from the driving vector.Analogously authors train a mapping from 0.2 s sound excerpt to the driving vector (bottleneck features of the driving net). The resultant driving vector can be used to transform the source face in the same way as described before.
✔️ Experiments:
The model is trained on VoxCeleb dataset on cropped faces of size 256x256 pix.
The model is compared to CycleGAN and Averbuch-Elor et al. [1]. The proposed method obviously produces better results than CycleGAN. Compared to [1], X2Face makes fewer assumptions about the input data (e.g. the
source face should not necessarily be in a frontal pose with the neutral expression, etc.) and can handle larger pose changes. In contrary to [1], X2Face can work with a single driving frame and does not require a driving video.