Get the latest tech news

The State of Generative Models

ble of Contents In the face of disruptive technologies, moats created by closed source are temporary. Even OpenAI’s closed source approach can’t prevent others from catching up.

This is essentially a stack of decoder-only transformer blocks using RMSNorm, Group Query Attention, some form of Gated Linear Unit and Rotary Positional Embeddings. In 2025, I think that it is likely that we see an Anthropic reasoner but also that this inference time compute paradigm can be scaled up alot more which gives the open-source community (and other labs) a lot of room to catch up. To quantify, I think we will hit 90% SWE-Bench Verified by the end of 2025 (o3 is supposed to be ~70) and translating that to real world tasks, models will probably be able (~50% maybe) to debug or implement a single feature within a larger code base.

Get the Android app