Get the latest tech news
Recent results show that LLMs struggle with compositional tasks
Recent results show that large language models struggle with compositional tasks, suggesting a hard limit to their abilities.
Binghui Peng, at the time a doctoral student at Columbia University, was working with one of his advisers, Christos Papadimitriou, and colleagues to understand why LLMs “hallucinate,” or generate factually incorrect information. Then, in December 2024, Peng and colleagues at the University of California, Berkeley posted a proof — without relying on computational complexity conjectures — showing that multilayer transformers indeed cannot solve certain complicated compositional tasks. For example, Tom Goldstein, a computer scientist at the University of Maryland, and his colleagues added a twist to how they presented numbers to a transformer that was being trained to add, by embedding extra “positional” information in each digit.
Or read this on Hacker News