HOW DO VISION TRANSFORMERS WORK?

Global and local aspects consistently show that MSAs flatten loss landscapes. Left: Loss landscape visualizations show that ViT has a flatter loss than ResNet. Right: The magnitude of the Hessian eigenvalues of ViT is smaller than that of ResNet during training phases. Since the Hessian represents local curvature, this also suggests that the loss landscapes of ViT is flatter than that of ResNet.

https://arxiv.org/pdf/2202.06709.pdf