Get the latest tech news
Language models pack billions of concepts into 12k dimensions
In a recent 3Blue1Brown video series on transformer models, Grant Sanderson posed a fascinating question: How can a relatively modest embedding space of 12,288 dimensions (GPT-3) accommodate millions of distinct real-world concepts? The answer lies at the intersection of high-dimensional geometry and a remarkable mathematical result known as the
In a recent 3Blue1Brown video series on transformer models, Grant Sanderson posed a fascinating question: How can a relatively modest embedding space of 12,288 dimensions (GPT-3) accommodate millions of distinct real-world concepts? "Archery" shares some semantic space with "precision" and "sport""Fire" overlaps with both "heat" and "passion""Gelatinous" relates to physical properties and food textures"Southern-ness" encompasses culture, geography, and dialect"Basketball" connects to both athletics and geometry"Green" spans color perception and environmental consciousness"Altruistic" links moral philosophy with behavioral patterns Sphere Packings, Lattices and Groups by Conway and Sloane Database-friendly random projections: Johnson-Lindenstrauss with binary coins by Achlioptas Hadamard Matrices, Sequences, and Block Designs by Seberry and Yamada
Or read this on Hacker News
