Get the latest tech news
Konwinski Prize
Why I launched the K Prize, aka, why want to give $1M to the first open source AI that gets 90% on contamination-free SWE-bench.
Even if SWE-bench didn’t officially publish the test set, the benchmark is composed of issues and code scraped from public GitHub repos, and most models today are trained extensively on those same repositories, so contamination is a likely happening. I came to appreciate the power of programming competitions to catalyze research progress back at UC Berkeley, where I witnessed the Netflix Prize (which was also $1M and based on real data) inspire my friend and Databricks co-founder Matei Zaharia to create Apache Spark. And finally, shoutouts to the core members of the K Prize team for standing this up: Chris Rytting, Justin Field, Alex Shaw, K. Tighe, and Lindsey Gregory.
Or read this on Hacker News