Get the latest tech news
NanoGPT: The simplest, fastest repository for training medium-sized GPTs
The simplest, fastest repository for training/finetuning medium-sized GPTs. - karpathy/nanoGPT
If you are not a deep learning professional and you just want to feel the magic and get your feet wet, the fastest way to get started is to train a character-level GPT on the works of Shakespeare. We'll also use a much smaller Transformer (4 layers, 4 heads, 128 embedding size), and decrease the number of iterations to 2000 (and correspondingly usually decay the learning rate to around max_iters with--lr_decay_iters). For an example of how to finetune a GPT on new text go to data/shakespeare and run prepare.py to download the tiny shakespeare dataset and render it into a train.bin and val.bin, using the OpenAI BPE tokenizer from GPT-2.
Or read this on Hacker News