Get the latest tech news

What is the best harness for each model?


FrontierHarness Eval v1.1 ran 22 coding-agent harnesses with 5 models on the same 62 tasks, 4 times each: 27,280 runs. The best-value harness was different with each model, and cost ranks moved far between models. With the model fixed, the most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest.

None

Get the Android app

Or read this on Hacker News

Read more on:

Photo of inference

inference

Photo of spend

spend

Photo of coding agent runs

coding agent runs

Related news:

News photo

DeepX claims inference up to 100B parameters at less than 5 W

News photo

A primer on key AI LLM concepts — KV cache, weights, quantization, and how inference actually works

News photo

A search-and-inference database from scratch in pure Zig