China trains a 10billion parameter multimodal network… using NVIDIA’s code:

A hybrid team of researchers from Alibaba and Tsinghua University have built M6, a “Multi-Modality to Multi-Modality Multitask Mega-transformer”. M6 is a multi-modal model trained on a huge corpus of text and image data, including image-text pairs (similar to recent systems like OpenAI’s CLIP). M6 has a broad capability surface and because of how it was trained, you can use M6 to search for an image or vice versa, generate media in different modalities, match images together, write poems, answer questions, and so on.

📦 Data: ~60 million images (with accompanying text pairs) totalling 1.9TB (almost twice the raw size of ImageNet), plus 292GB of text.