Get the latest tech news
Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires
're going to look at three different kinds of benchmarks, one set of calculations for baseline numbers for performance "napkin math" estimates, one set of AI model evals, and one on car tires. To build my intuition for things, I like thinking about them before seeing the explanation, so these are presented with the benchmark information first and the explanation later in case you want to think about your answer before seeing my thoughts.
None
Or read this on Hacker News