Get the latest tech news
Quadrillion-parameter transformers and how to train them
TL;DR Extrapolated to 5T tokens, our law prescribes ~7,000× Chinchilla's parameter count at 1029 FLOPs, and the count keeps growing with compute. Bigger models on repeated data need much more weight decay.
None
Or read this on Hacker News