Open source 2.7 billion parameter GPT-3 model was released

As you probably know OpenAI has not released source code or pre-trained weights for their 175 billion language model GPT-3. Instead, they decided to create a commercial product and exclusively license GPT-3 to Microsoft.

But open-source enthusiasts from eleuther.ai have released the weights of 1.3B and 2.7B parameter models of their replication of GPT-3. It is the largest (afaik) publicly available GPT-3 replica. The primary goal of this project is to replicate a full-sized GPT-3 model and open source it to the public, for free.
The models were trained on an open-source dataset The Pile which consists of 22 smaller, high-quality datasets combined together (825 GB text in total). Larger GPT models (10B and full GPT-3) are not available yet, but soon will be.

The 2.7B model likely won't fit into the Colab TPUs memory, and you may have to get some larger TPU pods to finetune from it. Sampling from it, however, works just fine.

It is the first time I tried some kind of GPT model, and I was not too impressed (see my fav results below). Maybe 2.7 billion params are not enough yet to leave the copyrighters without their jobs 🤷🏼‍♂️. Let's wait for a bigger model.

⚙️ Code in Mesh-Tensforflow
📦 1.3 and 2.7 model weights and configs
👫 Colab notebook