#jupyter_notebook #gemini #large_language_models #llm #openai #training #transformers

This project shows how to build and train a transformer language model from scratch in PyTorch. It uses the Pile dataset, tokenizes text with tiktoken, and stores tokens in HDF5 files for faster training. The code includes attention, MLP, transformer blocks, training, saving, and text generation. The benefit is that you can learn how LLMs work and train your own small or large model on a single GPU, then use it to generate text for your own tasks.

https://github.com/FareedKhan-dev/train-llm-from-scratch