Get the latest tech news

Can Pictionary and Minecraft test AI models’ ingenuity?

Some AI enthusiasts, in search of better AI benchmarks, are turning to games like Pictionary and Minecraft.

Calcraft was inspired by a similar project by British programmer Simon Willison that tasked models with rendering a vector drawing of a pelican riding a bicycle. In contrast to text-based benchmarks, games provide a visual, intuitive way to compare how a model performs and behaves, said Matthew Guzdial, an AI researcher and professor at the University of Alberta. “I think the good qualities Minecraft does have from an AI perspective are extremely weak reward signals and a procedural world, which means unpredictable challenges,” Cook continued.

Get the Android app