CvT: Introducing Convolutions to Vision Transformers

Another improvement for Vision transformers! Inject inductive biases of CNNs (i.e. shift, scale, and distortion invariance) to the ViT architecture while maintaining the flexibility of Transformers.

❓How?
Main architectural novelties:
- Hierarchical architecture
- New convolutional token embedding
- Convolutional projections before self-attention instead of the linear which was used in ViT. This is where convolutions come into play.

✅ Results:
Almost SOTA on Imagenet 1K and 22K: 83.3% and 87.7%.
Almost because Swin Transformers with local window self-attention layers and downsampling layers are a bit stronger (see the image with results) and perhaps faster.

🤔 Looks like it is a trend now to incorporate useful structural properties of CNN into Transformers. I'm pretty sure, we will see more papers like this in the next few months.

📝 Paper arxiv.org/abs/2103.15808