Get the latest tech news
100M Token Context Windows
Research update on ultra-long context models, our partnership with Google Cloud, and new funding.
By learning to recognize the unusual nature of the “needle”, the model can ignore otherwise relevant information in the “haystack”, reducing the required storage capacity to less than it would be on real tasks. It’s several orders of magnitude smaller than frontier models, so we would be the first to admit that its code synthesis abilities were not good enough yet, but it produced the occasional reasonable output: “The current and future impact of AI is fueled to a great extent by the development of increasingly capable large language models.
Or read this on Hacker News