Get the latest tech news

Test/verification technique effectiveness with Astra


When we looked at how well agents used test/verification techniques in August1, we saw that GPT-5.6 Sol generally did what you might call a caricature of various techniques when asked to use a technique or library such as TDD, QuickCheck, Lean 4, etc.; not specifying any particular test method (which we'll call Default) resulted in many worthless unit tests, which was one of the best performing conditions because asking agents to test in specific ways mostly made them worse. In that eval, we gave agents the Zstd RFC and asked them to implement Zstd (with an addendum for how they should test for all conditions other than Default).

None

Get the Android app

Or read this on Hacker News

Read more on:

Photo of Astra

Astra

Photo of test

test

Related news:

News photo

Siri Glitches Ruin Reviewer's Test of Apple's New 'HomeView' Smart Speaker

News photo

Do House members' stock trades beat the market? A test of 8,900 trades

News photo

Nvidia Fallout Puts China-US Trade Truce To The Test