👆🏻BYOL - Bootstrap Your Own Latent
BYOL is a new approach to self-teaching image representation with 2 neural networks that interact and learn from each other. The online network learns from the representation made by the target network on the same image with various additions. The underlying BYOL architecture is existing ResNet50 or other similar architectures. Input x is padded to t and t ', which are transmitted via the online and target network separately.
The difference between online and target networks is that the former has an MLP architecture with two fully connected layers, and Relu and batchnorm in between. The online network view learns from the view generated by the target network. The online network is updated with a regression loss function whose targets are set by the target network. And the parameters of the target model are updated by the exponential moving average of the online network, allowing you to process more information and avoid decision collapse.
The performance of BYOL is in line with the comparison with the supervised learning architecture of SOTA. There is a slight performance degradation when using only random cropping as image enlargement, but BYOL performs better than SimCLR by iteratively learning from previous versions of its output without using negative pairs with the linear classifier protocol. However, the BYOL approach is not yet applicable to the tasks of processing text, video, and audio.
https://www.youtube.com/watch?v=YPfUiOMYOEE
https://ai.plainenglish.io/byol-bootstrap-your-own-latent-dacee62a3dc8
https://arxiv.org/abs/2006.07733
https://arxiv.org/abs/2010.10241
https://github.com/lucidrains/byol-pytorch